Would you charge someone for a misunderstanding?

Last Wednesday I wrecked my car.
I have been house sitting for a woman who left me her car keys and said I could drive her car. Typically even when I have permission I do not drive other people’s cars because of liability issues, etc. But because my car was not driveable for a couple of days I went ahead and borrowed her car – she already said I could, so no big deal, right?
Wrong.
Apparently I left some paper or something in the car, and when she got home she noticed I had been in the vehicle and was very upset. She was already in full-on chew out mode so I just listened and remained apologetic. I didn’t bring up the fact that she had told me it was okay. I felt that it would have made things worse in some way, so I just kept my mouth shut about it. When its your word against theirs, what good is it anyway?

She told me that her and her husband decided that they were going to charge me for the “wear and tear” on their vehicle, and took 150 dollars out of my pay for watching their house. I was very thankful for the use of their vehicle. And I feel super bad the wires got crossed somehow about it. If I had known she didn’t want me to drive it at all, I wouldn’t have done it. But I SWEAR she said I could.
I returned it to them in exactly the same condition it was in when I started using it. Same amount of gas and everything. I understand why she would be upset with me, that’s a pretty big misunderstanding. But a misunderstanding worth taking over half of my pay about? I don’t think so. What do you think?