Why don't nuclear physicists reign supreme over everyone with nuclear arsenals?

There are maybe a few thousand men on this planet who know how to really do some fun stuff with atoms. Why don’t these small groups of men play along cheerfully until they are given everything they need to create weapons that would make us all slaves to their wills? Benevolence? Scientists are incorruptible? Are the braniacs too weak? If they’re so darn nice why do they hand over their work to the guy’s with the itchy trigger fingers?