Do you believe there are limits in comedy?

I believe in free speech and expressing our ideas. I also often enjoy comedy. But when it comes to some comedians, I feel annoyed by their sence of humor because I feel that they are ridiculing some people all the time, they are being offensive for no reason and this makes me sad about them. I heard some people defending them because comedy is something made to make people laugh and attacking some people made others laugh. I believe than both sides have a point and I respect every opinion. What do you believe? Are there limits or not?