Should I feel guilty for traveling?

My husband and I have an opportunity to travel to Ireland (where my family comes from) and England for at least a week. I’ve been hemming and hawing about going. My husband thinks that it’s because I’m feeling guilty.
The rest of my family are struggling with finances. I’ve helped them financially at times. We’re not rich, just we’re smart with the money that we do have. I do feel a little guilty when I can do fun things with my immediate family and the extended family makes comments about it.
I really want to go, but I don’t want to put anyone out. How would you feel in this situation?