Why are most people afraid of ghosts?

Of course this only applies to people who believe in ghosts. Most of the time if you hear a ghost story, or hear someone say they have seen a ghost they are terrified.
I have never heard of any story (aside from urban legends) where someone has actually been injured by or even killed by a ghost. In fact, in most of these stories it is really just that a person sees something that they can’t easily explain, so they assume it is a ghost, and that is the end of that. So then why do people think ghosts are so scary?