Shouldn't wise guy and wise man mean the same thing?

I mean when they call you wise guy it’s usually a bad thing such as when someone says something and you come up with a witty comeback they say “okay wise guy”. But when you say something really smart they’d call you wise man.