Is it offensive to say "sorry" when you don't have any cash on you when approached by the homeless?

Had an experience with a disgruntled homeless person who asked me for spare change. I politely responded
“No sorry I’ve no money on me.”

Which most homeless people tend to understand to mean that I’m sorry I can’t help them. But this individual responds by saying.
“You don’t need to be sorry. What are you sorry for? What’s wrong with people?”

Honestly I just walked off, because I got the impression he may have been looking for a confrontation. But have things become so pc that the homeless are getting offended by the word sorry? Because I feel that by simply saying “No” kind of comes off as uncaring.