How to break it to him other than the fatal "We need to talk"?

So I have found myself in this very situation , having to break up with someone I wasn’t even going out with. We had been going out for about three months and had become very close, but when my trip to France approached at the beginning of the summer, he became very distant, and then explained he couldn’t be there for me anymore, he wanted me to be free to live my life, maybe meet someone else, etc. It seemed pretty clear to me: I figured he didn’t want a long distance relationship for two months, and I thought it was an amicable parting; I even thanked him for releasing me without breaking my heart. We spent a few days together in a platonic manner before my trip, and I thought we were already to the stage of “just friends”.

Turns out, he explained himself very badly. He recently told me that he had wanted to “back off” a bit because we were becoming too attached, but he expected it to be “just a break” while I was gone. He never said any such thing before, and I would never have agreed to “take a break” if I had understood what he intended. I was certain that he had broken up with me, and accordingly got over my feelings for him. But he is still in love with me, he keeps making sexual references which make me uncomfortable, and trying to kiss me. Frankly, I am not in a place where I want to be with anyone right now, and I can only regard him as a friend.

I tried to explain a few nights ago how I perceived the situation. I told him it can’t be the same as if nothing ever happened, and that I don’t want to have sex with him or anyone else right now, and he told me that made him sad because I’m the only one he wants, and that I misunderstood what he meant.

Does anyone have any thoughts on how I should explain this to him in a way that he will get it? Would a letter be ok or should I really do it face to face?