Is it rude that I use a napkin or toilet paper to touch bathroom handles in public or other people's houses?

Last week I went over my mothers friend’s house and I asked to use the bathroom. After I washed my hands and everything, I tore a peice of toilet paper off the rack to open the door to leave when the lady’s husband saw me. He asked me what I was doing using toilet paper to open the door and I told him it was because I like to since a lot of people don’t wash their hands after using the bathroom. He got offended and asked me some more questions about it. Then when I got home that night, I found out he told my mom about it and she told me it seemed rude to other people and not to do it anymore. Personally, I don’t think it’s really that rude but whatever.