How do you feel about couples where the female is taller than the male?

I honestly want an opinion on this. It seems like people look down on it and I want to know why it even matters? My boyfriend is significantly shorter than me (by four inches) and when we go out I see people rudely stare at us every now and then. It upsets me so much because we both have similar personalities and get along really well with each other. However, the looks still make me feel very self conscious. It already took me a long time to be able to accept my height (I’m 5’9”) with plenty of people telling me throughout my life that I was too tall for a female. Even though my boyfriend is short, he is the strongest and most intelligent man I know so why should that be looked down upon? I am even ashamed to say that it took me a long time to date him because I was very fearful of the looks I would get, not because I didn’t like him.