Why do some people have a more difficult time adjusting to the change of seasons than others?

Pretty much said it all in the subject.
I have always lived in a place that has a wide variety of weather, but I feel like every time I finally adjust to the seasonal weather, the season is about to change again. At the same time, it seems that most of the people around me are comfortable with the temperature long before I am.

Are there certain factors that contribute to this, or is it random? Is it just an issue of perception?