Should I let him go?

I have been dating someone for four and half months. Everything seems great, but he is terrible at communicating. The chemistry is great, and when we are together, it is awesome. He is a salesman, and lives 2 hours from me, so we only see each other 2–3 times a week. He has a troubled past… terrible childhood, terrible first marriage, and now, he seems unwilling to commit! He tells me he is selfish, and I am crazy for putting up with him. I know there is no other woman (trust me, I have explored and investigated to my satisfaction), but he recently told me he simply does not want to live. It is as if he is terrible at relationships, so he throws himself into his career… (he is very sucessful). I know he sounds crazy, but I don’t think he would commit suicide or anything, but he simply has not found the joy in life. I kind of feel sorry for him, but know that I can not do anything! I have tried the “Break up” approach only to be reunited with him. Then, he tells me how happy he is I am in his life, but he does not change (I don’t want him to change who he is, I just want him to try, and give us a chance. Am I CRAZY for staying with him? Should I let him go? I have tried going out on other dates, but my heart belongs to him! I am thinking about him! I just broke up with him again, and I simply did not respond to his last 2 texts, he then “quit”... but then I “dropped him a line” (text), and he responded. I am hopeful… am I hopeless? My friends say he is WEIRD… and that might be the case, but what do I do?? I have not met his family, and he really “backed off” of the relationship when it got (on the verge) of serious. Is he scared? or am I stupid? I just am in love, and I feel he cares, but he can’t get over this. My sister told me to “let him go” and if he comes back (like the saying says), he was mine, if not, it was never meant to be…
Well, I’m scared he will simply let it go, and think I don’t care (but he KNOWS I care tremedously for him)... I actually love him. HELP!! (I am 25 years old, he is 30) He says he is selfish and likes to do what he wants… and all relationships in the past have been failures… he is afraid to fail again, but his past relationships (ex-wife and ex-girlfriend) have been immature… I accept his career (respect his work ethic) and simply adore him! what do I do???