If caffeine doesn't have an effect on someone (as in, they can drink a whole pot of coffee and then sleep an effortless 8 hours), then is excessive caffeine still bad for that person?

We were talking about a dispatcher here who makes a pot of coffee about 9 o’clock every night, and I suggested that maybe caffeine doesn’t wake him up, because I am also unaffected by it. It stands to reason that he’s addicted to caffeine since he downs coffee all day, but I rarely drink any caffeinated beverages, so I am not addicted, but I am still unaffected by it.

Does caffeine still bear the same unhealthy aspect for people like me? Is it the rise in blood pressure and whatever else wakes you up that makes it unhealthy? Or is there something else to it?