Why does the use of the word "fag" by gay people seem to not bother other gay people?

My roommates are a “married” gay couple. One of them uses the word “fag” more than I’m comfortable with. The first time he said it, it startled me, and every time he says it, I get a little bit rattled on the inside. I’ve lived there 2½ years so I’m as used to it as I’m going to get.

But I’ve seen him do it in the middle of a party or a crowded room, and from my observations, only the straight people were surprised or showed any kind of “blip” in their facial expressions or whatever you want to call it.

I don’t understand why that word doesn’t seem to offend gay people if a gay person uses it; why I’ve only seen the straight people react to it.

I don’t mean to generalize; I am just commenting on what I’ve experienced in my own house for nearly 3 years