What should I do about a battery that is leaking white stuff into my favorite little clock?

I would make a paste with baking soda and water. Use that to try and break down the white stuff (use sparingly). The white stuff is battery acid and it is best to avoid touching it. Rubber gloves are your friend.

I bet it’ll pop right out if ya stroke it real nice…

Well, I made a paste with baking soda, put on rubber gloves (which I never would have done), and then pried the old battery out. Cleaned it out with baking soda, put a new battery in, and good as new. Thanks for the advice, @johnpowell.

Get something with a small sharp tip and gently scrape it off. I had a flashlight that wouldn’t work because acid had leaked onto it and once I scaped it off and put fresh batteries, it was working again.
Hope it works for you.

It’s corrosion. <—This link goes to advice.

Hey now, hey now hey now, sing this corrosion to me…

Sounds like you already solved the problem. I’ve dealt with this situation many times. After cleaning out the gunk, use an emery board to restore the bare metal contact points where the new battery goes. Most of the time the device can be restored with no real damage.