Is it inherently racist to be worried about the continuation of one's own race?

Whatever you might personally think about race—that is, regardless of whether you think it reflects real taxonomic differences between people, is merely a social construction, or something else—some people find it important. One contemporary argument against miscegenation is that, while there’s nothing wrong with other races, everyone has an interest in making sure that their own race survives.

Personally, I could care less about races surviving—whatever they are—so long as their rise or fall comes about naturally rather than as the result of genocide, oppression, or the like. What of those who do care, though? Does the fact that they care make them inherently racist, or are they quite within reason to be worried about the future of their race?

In answering the question, please try to set aside the fact that most of the people who give this argument do not actually believe that there is nothing wrong with other races. That may be true, but I am wondering what people think of the argument itself and what it reveals about those who give it.