Would you move away?

I currently live in NY state with my husband and 3 kids. We don’t love it here. Winters are long and cold, property taxes are very high, and our neighborhood is less than peaceful. We would like to move to a place with warmer weather, lower taxes, and more land. We have looked into North/South Carolina, and Tennessee. The one thing that keeps holding me back? Family. My husband is okay with leaving his behind but I have a grandfather who is very close to my children and he’s like a father to me. He sees the kids a couple times a week and it would be a big adjustment for everyone. Is it selfish to leave? It would be great to live somewhere warm and private, and actually be able to afford our home but at what expense? Have you moved away from family? Was it worth it? Did your family resent you for it?