Why can't people from Western countries hear "tones" in tonal languages well?

I have taken Chinese for a while now, and am able to hear whether the tone is neutral, goes up, down and up or just down pretty well, but most Westerners who I know have taken Chinese or another tonal language usually can’t hear it well. Why is this?