Under what circumstance is it acceptable to use the opposite gender's toilet facilities?

When is it okay to step into the ladies room, men? Women, do you scoot over to the men’s if the line is out the door at a concert?

Maybe a roadside gas station that clearly doesn’t get a lot if action is less stringent with the signs on the door. What if your gender’s facilities are filthy by comparison to the one just next door?

Does it bother you when you see the opposite sex in your space? Is it really a big deal?