Should I stop wearing this hat if people call me out on the street?

I have a cute bunny hat that I absolutely love to wear in winter. I’ve had comments from men on the street like “Stupid hat.” or “Weirdo.” They don’t laugh, they look so disturbed by my hat which is very rude.
Latest I got was a group of men in a car pass me by and shout that I look stupid. Sometimes I get intimidating looks from females.

My style is sort of like this if I am absolutely confident: http://www2.pictures.zimbio.com/fp/Ezra+Miller+Arriving+Toronto+Airport+hwispw2tTk9l.jpg

I once wore pjs with sheep on them once to school and back without second thoughts. I am not dirty, but I am comfortable like this and I feel like myself wearing baggy clothes like the picture. I feel complete.

I’m not keen on stopping but I wanted to know your opinions. Would YOU stop? Why should I or should I not stop?