When someone refers to something as "the former" or "the latter", do you have to stop and think?

Firstly I have to figure out which one is which (I figure it out by knowing what “former” means… so if it is “latter” then it is the one that isn’t the “former”), then I have to re-read the sentence to figure out which option they were talking about in the first place.

Is it just me?