Why do people equate "comfortable" with "over-sized"?

I hear it all the time, and I don’t understand.
When [most] people say they want to be comfortable in clothing, they will put on the biggest, baggiest, sloppiest clothes they have. Why? Or they use comfort as an excuse to dress sloppily in over-sized sweats and big t-shirts. I don’t get that either.

Do you do this?

I suppose you can tell that I don’t. I’ve never found any clothing that I was wearing to be uncomfortable. Tight dress, tight pants, hose: I’m comfortable. The only excepting would be high-heeled shoes because I’m overly-sensitive to pain, but that’s it. Even then I still wear them and carry flats in my purse. Actually baggy clothing bothers me because it gets caught up in itself when I move around.

Feel free to share your clothing preferences =)