Why does a bar of soap end up different from what it started as?

If you are cheap or lazy enough (a little of each in my case) and you let a bar of soap get really small, what you have at the end is much different from what you started with. The color is kind of icky, it is hard and brittle and does not lather as well. Do the manufacturers put this stuff in the middle of the soap or is this just a residue of material that does not dissolve as easily?