Does a troubled past affect your reaction to betrayal?

My partner has recently stated that my reaction to him betraying me with his female friend would have been different if I did not have such a troubled past i.e. I was in foster care due to abuse throughout my childhood. He stated that if I did not have past issues and trust issues with people I would have accepted his betrayal and allowed him to continue meeting his female friend, and allowed him to go round to her house for brews. He expressed that my reaction was ‘psychotic’.
To condense a history of my relationship with my partner…he was living initially with his now ex wife and continued his friendship with his female friend who he met daily taking his son to school. I was uneasy about his friendship with this female as he was also having a relationship with her until he met me. He continued to meet this female in the mornings and have breakfast with her and occasionally their sons had tea at her house and he remained in the house also. He consistently reassured me that there was nothing more going on between them both and that he didn’t fancy her, and that I had to trust him.
Four months into our relationship my partner experienced a bereavement and he confessed as I had strong suspicions…call it sixth sense, that over a 2 week period he slept with his female friend 3 times and once with his now ex wife. My partner confessed to his betrayal 6 weeks after the incident and to justify his betrayal he claims that his head was ‘all over the place’ and he wasn’t sure if he wanted to be with me. One of the incidents with his female friend (which I have only found out recently) occurred when his son was at her house for tea and they fornicated at the top of the stairs whilst 3 boys, including a 2 year old were left unsupervised downstairs.
As I did not want to be controlling I agreed to meet my partner half way and asked that he only meets this female in public. However following ups and downs in our relationship of which I know he confided very personal details about our relationship and I also saw evidence of her flirting on msn with him, and in text messages, and requesting that he goes to her house, stating reasons such as asking him to do some wiring etc…, I no longer tolerated their friendship.
The incidents occurred almost 2 years ago but I am unable to tolerate their friendship and definitely don’t trust them to be together, not even with children in the vicinity. I have to occasionally tolerate her bringing her son to our house for tea and she hangs about in our home with her son and then takes him home. This distresses me greatly and I feel my partner is putting me in a situation that causes me anxiety and brings back all the hurt that he caused me.
My partner claims that my reaction to his female friend would be different if my past was more settled and that I would continue to tolerate their friendship. I stated that if my partner and I had sought relationship advice to address my ‘trust’ issues they would have advised that the female was totally removed from our relationship if he wanted me to regain trust in him. What do you think? What is the general consensus on a how a person would react to my situation?
P.S I am very hurt by my partner’s statement as I have never played the victim regarding my past and I am proud of who I am and what I have achieved.