Is a man not wearing his upper clothing a taboo in western society?

Not a particularly fascinating nor insightful question I shall admit but my discussions with some friends regarding the subject has encouraged my curiosity.
I spoke to a group of friends (some straight males, some straight females and a homosexual male) who, for the most part, agreed that when a male is not wearing their upper body clothing it generally implies that they have a vain and unlikable character, even if the weather is especially hot on that day. What is everyones thoughts on this phenomenon? Do you assume males who are dressed (undressed?) in such a way are uninteresting or undesirable people or is your initial thought that they are simply too warm?
I, for one, would like to remove my t-shirt outside on a hot day (so three times per year) if possible but feel I would be self-concious not of my body but of the fact that I would assume others are judging me negatively for such. Does anyone share this sentiment?