What is the professional way to deal with colleagues?

Generally I’ve been hanging out with three colleagues at work. We go to lunch together everyday and sometimes we give a coffee break etc. Last week, one of them told that she was going to toilet with loud noise which was obviously for me. The other one just left the room. Another one just made a murmuring sound and told she felt like having some coffee. This was her way to use another door to go outside secretly. I mean this bothered me because there is no need to fool me just to hang out without me. I really dont care whether they invite me or not. But this is impoliteness. I mean it is very easy to break the connections but it is unprofessional. By the way I am their supervisor, their tricks only made me think they are just trying to get along with me because of that. I am thinking of not having lunch with them anymore. But this action makes me look childish especially when I am right. But I felt like tricked and want to give them a lesson.