What explanation, if any, should you give when declining a job offer?

If you’ve gone through an arduous and highly selective interview process, and have succeeded in getting a job offer, how do you turn it down if there are specific, negative reasons for not wanting the job? For example, if the geographic location doesn’t suit you, or if you discover that a potential future boss (or an important colleague) would be horrible to work for. Should you be honest and explain your reasons, diplomatically, or should you simply give a vague reason?

Note that this is a highly coveted job and a fantastic career opportunity, so the organization will want to know why you aren’t interested. There is also the consideration that since your line of work consists of a fairly small community (and everyone talks to everyone) that you don’t want to upset people that you’ll interact with in the future.