How would it look to a potential employer if I didn't allow them to contact previous employers?

I’m just curious about this. I know they can’t say anything bad about my past preformance or anything. I’ve had disagreements with management before and quit because of that. I’d prefer that they not contact them because I’m not sure if it is going to turn into a he said/she said kind of deal. This is only for a couple past jobs, not all of them. I’m just wondering how that would look to a potential employer. I’ve nothing to hide, maybe just a slight paranoid. I haven’t worked in a while.