Why do we say "he's got balls" to imply that someone (male or female) is tough?

I am confused because… I have balls. I know how vulnerable and sensitive they are. I don’t like them tampered with (too much) and I certainly wouldn’t use them as an all time symbol of my unflinching manliness, because the truth is a simple flick of the finger in a strategic position can send me crawling on the ground asking any deity that will listen to end my life quickly, so why would the most vulnerable part of the male body be used as a symbol of tough courage?