How do you dump an activity partner?

I’ve gotten in the habit of working out with a friend. We went together randomly a couple times, and now it has become an almost daily thing.

Sometimes he kind of gets on my nerves, like he’ll invite himself over after we work out, or he’ll push me too hard when we work out. But now I’d feel guilty if I just went to work out without him and didn’t say anything. Do I just tell him that he’s annoying sometimes? What’s the best way to handle this?