Why is it considered racist to express a physical attraction to people with a certain skin tone but perfectly acceptable to be attracted to someone based on other physical attributes?

I responded to this question a few times and had an interesting exchange with another user about race and how race discussions are always messy. This is true, but what I want to know is why are discussions of skin tone messy in regards to physical appearance?

It’s perfectly reasonable for someone to pick a prospective partner with things like hair color, height, weight, body type and even eye color in mind, but as soon as someone mentions skin color, they are racist. Why is that? Why is skin tone not an acceptable physical criterion for choosing a potential mate?