Questions on relationship (determination) faux pas. Which of the following (do you think) is worse? [Details inside].

In about six months time I managed to ‘insult’ a male co-occupant of our apartment building, twice (two different men).
Last week this guy and I were together in the elevator when he said to me that it had been a while since we had seen each other.
I answered that it had not been a while because I saw him a few days before, together with his mother.
He answered, smiling awkwardly, that in fact that was his wife.
A few months prior the same blooper happened, with another guy.
Again, I confused his girlfriend to be his mother….
Would the opposite faux pas be worse “I saw you walking with your wife.” “No, that is my mother.”
Would you be insulted if one of these two happened to you?
Can you think of other relationships (judging) faux pas?