Can someone help clarify what is believed to be how time travel would work and why that is believed?

Imagine I travel 50 years into the future to 2062 a future that didn’t have me to influence it for 50 year, I think we would all agree that future would be different than a future were I didn’t decide to go into the time machine and maybe had a wife and kids.

This is because the future is set by rules and systems and if I was God and could pluck someone out of existence the future would be different as the system loses me and how I would effect everything within the system.

But we dont seem to think this way when thinking of traveling back in time. The past when traveling on the time line like we would to get to the future I dont see why it’s just not the rules but in reverse.

So if we turned back time 50 years ago it would be at some state from now when the rules happen in reverse, but if some 60 year old man went in a time machine plucked himself out of the universe lets say the rules wouldn’t give the same result.

Just like traveling forward in time it matters if you are there as the rules give different results, why is this not how people see going back in time?