What is the reasoning behind gambling being illegal on US soil?

I have never understood this. Gambling doesn’t physically hurt anyone and while it can inspire other illegal activity stealing, conning, etc. it does not directly cause other illegal activity.

Can the collective enlighten me as to why gambling is illegal in the US?