What is so wrong with being an elitist?

Why is it considered wrong to be an elitist? I mean, if you work really hard for something, why shouldn’t you be? Whether that’s physical, mental, educational, etc. Is it really that wrong? To think you are better than other people, who are lazier than you? Why does it have such a bad connotation?