Oh, you would not kill anyone, but who would you let die if you had the means to save them?

Many people say they could not out and out kill anyone, but I wonder if they would allow someone to die if they had the means to save them. For instance the building is burning, they are on the ledge all you have to do is stand the ladder up and they can climb down to freedom, who would you not do it for? Or what if they were in the river struggling against the current, about to go under at any second, you have a long enough rope, who would you not throw it to? By knowing you could save them and not, how do you justify or reconcile morality to your ”Golden Rule” by not saving a life you could have?