Can I use the phrase "black man" in a story without being thought of as a racist?

I’m writing a story set a little in the past and it mostly involves just two people, an old man and a waiter. But I need this other person to arrive in the story and I thought that in order for others to view this story in a way that the old man sees it I would describe the person coming in as a “black man” as he would not be familiar with the term “African American” or anything similar. So, would that label me or the story as being racist automatically in the minds of the reader or would it help with the story’s “ambiance” if you will? Thoughts? I thank you for your time.