Frustrating Habit?

There is a person in my life whom I love dearly, but we keep having this same issue. Something will happen that does not feel right, and when I confront the person with it the response is “well it would not bother me if you did it.” To me this seems wrong because it is letting the other person off the hook for having to consider that someone else might not view the behavior in the same way. Example, we were at lunch, a “finally catch up because we’ve been so busy” lunch, and I am told, “oh I am going to make a call midway through, you don’t mind right?” I asked, “is it an important call or can it wait?” The response “I don’t know.” My reply, “Maybe you could find out, and if it is not time sensitive we can enjoy our meal in peace.” Reply, “It wouldn’t bother me if you did it, so why are you making such a big deal out of it?” Given that we were meeting to catch up it felt wrong for someone to just leave in the middle of the lunch to chat. The same behavior happens in other situations too where things will be done without considering how I might feel because it would not bother the other person if they were treated that way? Am I being too sensitive or is the other person being a bit presumptuous and rude?