Does anybody know the origin of the phrase,"Nuts to that?"?

I find it an amusing phrase and would like to incorporate it into my everyday speech, but I’m worried that it might have NSFW implications. I try to keep my language PG.

I realize that creative people can give anything NSFW implications; that’s not what I’m worried about.