How do we know the universe is expanding?

So I was watching an episode of the universe a couple nights ago before sleep and this question seemed to be stuck in my head when I woke the next morning. Like I understand through red shift and what not we can see that galaxies are moving away from each other at extraordinary speeds. But I dont see how that really says that space is expanding.

I mean like say the universe looked like this billions of years ago. Then billions of years later it looks like this. Yea sure they expanded away from each other and there is a greater distance between them of unoccupied space than before but theyre still within that space.

Though now thinking about it I dont even think the circle should be there as I think space is infinite.

Really what Im getting at is how can we space is any bigger than it was at one point than any other if we cant actually see space at all, just the things in which occupy parts of it.