Is it polite to continue to speak in a different language when you have a guest?

Even though between us, my husband and I have 9 languages, we are having a fight about when it’s appropriate to use which language when you are a guest, or if you have a guest. For example, I don’t speak his mother tongue, and he doesn’t speak mine, but everyone in both our families speaks English well enough. If he or any members of his family come over, everything is in English. But if I go to his family, everything is in his language, and I am bored to death. I understand that it is natural to speak in the native language of the family – it’s easier and quicker, but if a guest is there, and they are sitting beside you on the couch, wouldn’t you want to include them in the conversation by using a language they could participate in? It doesn’t even have to be English! If his family comes to our house, at the dinner table it invariably goes to his mother tongue, and I feel like I am a stranger in my own home. I remind them to pick any language that we all can speak, and it lasts for five minutes. Finally I get up and leave. Am I wrong to feel like they don’t even want my company? I am getting really huffy. What is your opinion?