What makes religion, religion and a cult a cult? What separates the two of them? Why is one ok and the other frowned upon?

It seems to me that the only thing that separates religion from a cult is the amount of followers you have. What do you think? Why do we frown upon a small amount of people that follow their own set of rules yet praise (loosely speaking) those that are in large numbers?