Why are drones so scary?

I’m not sure I get the significance of the pilot being at a remote location. If a hellfire missile hits you, aren’t you just as dead when the missile is fired from a manned aircraft as from a drone. Maybe it’s like the radio announcer who botched a news report, saying; “Three people were killed in an accident on Main Street today, two of them seriously.” Are only drones able to kill you seriously? Can’t a crazed guy with a baseball bat do the job just as effectively as a drone?