Do mailmen have to deliver no matter what?

I understand that there may be circumstances, such as dangerous situations, but the situation I’m referring to is specific.
Our mailman refuses to walk over the baby gate at the end of our porch. We do not receive our mail if I forget to remove the baby gate. He won’t even leave it on the porch or garage, or anywhere.
I leave the baby gate up, because sometimes my young son opens the door and runs out, or opens the door and lets the dog out. Yes, I should lock the door, but I tend to forget The gate is not high, probably no higher than the knees on a man.
I was in the living room the other day, and saw the mail man looking at the mailbox, about to walk away with a handful of mail..so I ran out and got it, and he told me he can’t and won’t go over the gate.
If he’s too old to step over a gate, I don’t think he should even be delivering mail. Or is it on the property owner to always have a a perfectly clear walkway? (to me this doesn’t make sense since I’ve seen some hard to get to mailboxes before)