Is there a difference between "Uptown" and "Downtown"?

I have heard both (“I am headed uptown” or “I am headed downtown”). I would think they are interchangeable from how they have been used. But, being that up and down are opposites, wouldn’t you think that the words uptown and downtown are opposites, or at least don’t mean the same thing. Explanation, please! I’m confused!!