At what point does a difference in the size of your pupils become alarming?

About a half hour ago, I noticed that my pupils were different sizes. The left one was small, and the right one was larger and changing sporadically. I went away for a while, and when I came back, they were normal. A bit later, the right one was small, and the left one was larger and changing sporadically. I find this interesting because I haven’t had any head trauma or anything like that, and my head doesn’t hurt. Could it have something to do with the fact that I’m not wearing my glasses after a day of wearing them? I wear them all day, every day.

Needless to say, I’ll be watching it, more out of curiosity than anything. If it keeps up, at what point should I go to a doctor or optometrist?

Any theories that could explain why they’re behaving this way? Is it normal? Beyond head injuries, I’ve never heard of anything about pupil size.