Is it time to end things?

In December, the guy I am currently dating asked me to be with him. However, at the time, I was dating my ex. My ex and I broke up in January. In March, I told the guy I’m now dating I would like to give us a chance but we didn’t start dating until June.
I recently discovered he started dating his ex again in April but it didn’t last, only for a month. He never told me anything. When him and I met, they had broken up in August. But he has said they’ve been on and off since last year.
I’m extremely hurt that he went back to her in April after in December when he told me he wanted to be with me, I felt his ex is someone in the past.
Is it time for me to leave? I feel I end up getting hurt, that they would always go back together. I asked him if he still had feelings for her, he said “not really”. I said that means you do and he said no and that I don’t have to worry about anything.
I can’t just NOT worry. They’ve been together for 4 years and they are both 6–7 years older than me. Plus they see each other often as they work in the same field and has mutual friends. He said I don’t have to worry as it was their mutual agreement to end the relationship as it just isn’t worth trying anymore. I find it hard to believe as he went back to her in April when he said he won’t months before that.
Any advice would be appreciated.