What are the objections to the theory of memes?

It seems to me to be transparently obvious, but I have seen various objections which make little sense to me. The theory is fairly simple. There are three key components.

1. Ideas travel from one person to another. For those of us who are materialists, we have to assume that these ideas have some sort of physical configuration (meme) in our brains. If you are not a materialist, you have to at least accept that in some sense ideas exist inside of you or have become a part of you.

2. These ideas are parts of the colony of ideas in us. Individual ideas can be modified or perhaps rejected.

3. Ideas can improve our fitness.

We therefore have the three necessary and sufficient conditions for evolution – reproduction, variation and selection. Culture is the phenotype of memes just as out physical phenotype comes from genes. Good ideas will favor the health and social position of their holders, who will then spread these ideas to others.

You can say that we can use our power of reason to distinguish good ideas from bad ones, but in the final analysis the ideas that persist are the ones that have the most benefits. For example, we can argue over whether religion makes sense, but from a memetic point of view, it has been very successful, although, as a side note, I would say that religion is declining in industrial countries because it has outlived is usefulness, but that is a topic for another day.