Why is pointing considered rude?

If it’s not accompanied by laughing or jeering, I don’t really get why it’s such a big deal.

Is it seen as a physical extension of the gaze or something?

I’d want to know why someone was pointing at me, but I wouldn’t exactly think it’s rude. And I wouldn’t mind friends pointing at me either.

Do you care if strangers, acquaintances, friends or family point at you? Does it matter if it’s from afar or right next to you?