Are there any problems with slight pacifism?

I don’t know if ‘pacifism’ is the right word, but sometimes I get flak from people because I don’t feel the need to be aggressive in certain situations. I tell some testosterone-ridden friends of mine that I will not help them in bar fights if they start them, I feel I should not have to defend a s/o if she starts a fight with someone etc. I see no reason to escalate a situation or have a shouting match with someone over some trivial matter, if you can drop it and walk away, I will do it.

I have always seen fighting for any reason other than self-defense as ignorance (ironic, being in the military). Is this too naive or unrealistic, or is there a point to being aggressive sometimes?