I don't get it. Why are so many people bad at math?

This has always bugged me. I’ve been great at math for as long as I can remember (except for a period of time where I couldn’t do long division), but tons of people seem to just not get it.
Why is this? Is it because numbers don’t physically take up space, which leads to some complex thought? Or what? I don’t get it.