What does the phrase "with all due respect" really mean?

I find that when I use that phrase, I sometimes mean that the person I’m talking to is deserving of a great deal of respect and I’m afraid the comment will offend them. However, I also find that when I say that, sometimes I mean the person deserves little or no respect whatsoever and I’m only saying it to a) mock them and b) making nice. Are there more ways to use it? Is one of these ways more commonly used than the other? What is truly meant by this phrase?