How do I deal with someone who is obsessed with me?

Two years ago, I started going to an LGBT discussion group, where we talked openly about our problems, and I became friends with someone who went there too. We share an interest in classic movies, jazz, and old television shows. A few months later, we went out for lunch, and we both talked about being on the autism spectrum, him being more on the severe end.

A couple months passed, and during that time we went out to lunch, movies, hiking, etc, once every two weeks. Basic stuff that casual friends do. This was the time that I realized I didn’t have all that much in common with him, except for some superficial things.

Naively, I didn’t realize the implication that he might think I was romantically interested in him, because I definitely wasn’t. The next few months I played along, each day convincing myself that I was going to tell him I wasn’t interested in him that way, but that conversation didn’t happen for a few months.

A few months ago I finally told him that being casual friends is what I want. He completely broke down crying, verging on a tantrum, begging me to stay at his house until he was ok. The past few months he’s been asking me to visit so he isn’t lonely, and I visit him out of guilt.

So now, I feel like I’m being his friend simply out of guilt. And his autism makes it very hard for him to realize that I no longer want to be close friends with him. Plus, it seems like his obsession with me is growing stronger day by day. Today he texted me that he wants me to visit him as often as I can. And I just can’t do it.

What are your thoughts on how I should go about this?