I'm ashamed of the guy I'm seeing, am I right in thinking like this?

I’ve been seeing a man 16 years my senior for a few months, and so far I’ve told my friends he’s only 8 years older. I think I’m afraid they’ll judge me or think I’m crazy for dating someone who’s almost 40 when I’m in my early twenties.
There’s also the physical aspect where he’s not my usual pretty boy type; I love him for his personality and find him attractive but only because I know him, if I saw him on the street I probably wouldn’t look twice whereas I’m the type of girl who always gets attention from strangers.
I think maybe the imbalance in conventional attractiveness and the age difference is making me feel insecure and I can’t imagine how I’d introduce him to my friends. I guess I find this upsetting that we’re not the usual couple, but I couldn’t bear to leave him because of it.