Is it reasonable to be concerned about freak traffic accidents?

Every time I go through a green light I am tempted to slow down so I can check and make sure nobody’s going to plow through the red light and smash into me. I know I wouldn’t be at fault if that happened, but my passengers or I can still get hurt or killed even if I’m not at fault, so the thought still worries me.

I also know that I’m pissing off the people behind me if I slow down in front of a green light, though, so I usually don’t do it.

I know about defensive driving, but how defensive is too much? Are my fears unreasonable?