Is it moral to distance yourself from someone whose flaws were more than you could handle, if the flaws are not their fault?

For example, if you had a friend who was mentally ill and as a result tended to be very aggressive, and didn’t always direct his aggression towards the proper sources. You, as his friend, often receive the brunt of his aggression even though you don’t do anything to deserve it.

If it gets to be too much, is it ethically permissible to distance yourself from this friend? On the one hand, nobody should have to put up with abuse, on the other hand, it’s hard to feel okay about “punishing” people for things that aren’t their fault.

(Disclaimer: I don’t mean to imply that everyone with mental illness treats their friends badly.)