What evolutionary function does annoyance serve?

I’ve been painting today, and all day I’ve been annoyed and infuriated by a housefly. It seems to be in love with my sweat, and every time I forget about it, it bounces off my face like a trampoline or lands on my forehead for a drink. And as a result, I’ve ended up either painting my ear or making a Jackson Pollock painting as I swat at it instinctively, paintbrush in hand. The second I would put down paint brush and bucket and grab for the can of Raid, the fly would vanish and cackle evilly to itself as it waited for me to forget about it again.

I have to assume that I have evolved to find flies annoying because my ancestors who swatted angrilly at flies were less likely to get fly-borne diseases. But there are plenty of non-annoying bugs, and it seems to me that it would be in the housefly’s best interests to be the least annoying it can be and avoid being swatted. After all, no one says, “There’s just too many pretty lights flitting around my garden; damn those fireflies.” And when’s the last time you looked for a swatter when you saw a ladybug?

There must be some kind of evolutionary advantage to the housefly in being absolutely infuriating, especially given the prolific nature of houseflies; it wouldn’t take much time to produce less annoying behaviour if there was evolutionary pressure to favour such a mutation.