Do you REALLY care what people think of you?

My boyfriend is obsessed with image. I live in a small town and travel enough to realize this isn’t the whole world. I don’t care what the people here think (aside from him, my bosses,[whom I actually have respect for], and my children). This infuriates him, as he feels it directly reflects on him. It’s not as if I roam the town daily telling people to f*** off, or showing my private parts. I just say how I feel to a face and not a back. He hates that so much that he’s feeling we aren’t compatible. What do you think?