I rejected him and now he is mad at me, should I feel bad?

I went on 2 dates with a guy that I knew for about a week. We flirted and told each other that we liked each other (but not like, feelings).
But after the second date I started doubting if I wanted a relationship, but he made it clear that he wanted more, so I told him needed some time.
So 2 days later I tell him that I feel like we can’t be more than friends.
Because I wasn’t really attracted to him, but I didn’t tell him this part.
So now he feels used and he’s mad at me for giving him so much hope.
I tried to explain it and told him that I didn’t want to lead him on.
Now I’m feeling bad because of it, but I know I made the right decision. And of course we only knew each other for week.
Should I feel bad?
(sorry for my bad english)