What is this brown stuff I'm wiping off this old wooden desk?

So I’m getting some brownish residue whenever I rub a tissue or paper towel (or my sleeves) over this very old looking wooden desk. What is this stuff? Brownish residue is rarely good. My guess is that the varnish is coming off.