I found this to be a delightful and non-trivial mathematical puzzle. I’m a fan of such puzzles but don’t recall ever seeing this one before. Here’s my approach:

Each doll has a cup-shaped bottom half. Let the outermost of these cups have height **h** (measured from bottom of base to the top rim) and base thickness **b** (measured from bottom to top surface of the base). Each cup/doll is reduced in size from the previous by some fixed ratio **r** , 0<**r**<1. These 3 parameters (h, b, r) determine the whole setup.

Consider just the first two (outermost) dolls.

The second cup rests on the first cup’s base. The distance from the outer cup’s base to its top is (h – b). This must equal the total height of the inner cup (so their tops line up), which is r times h. [Sure wish I could post a diagram.]

rh = h – b

From this we can derive various relations:

r = 1 – b/h

h = b / (1-r)

b = h (1-r)

Given any two parameters, you calculate the third one using one of these formulas. This shows how they are constrained by conditions of the puzzle.

Example 1: Give a base thickness 10% of the height (of the bottom cup), then since b/h = 0.1 then r = 0.9. Each doll must be 90% the size of its predecessor. Example 2. If each doll is 80% smaller than the last, then the base must be 20% the height.

Alternate approach using infinite dolls instead of just two:

Bases of all the nested cups stack up on each other, forming a geometric series whose sum converges to some particular height. The final height of all the stacked bases will be:

S = b + rb + r^2b + r^3b + ...

S = b (1 + r + r^2 + ...)

S = b (1 / (1-r)) {applying formula for sum of geometric series}

S = b / (1-r)

But wait! The cups eventually get quite tiny, yet their tops all have to reach the top of the outermost cup. The only way this can happen is if the top of the innermost cup’s base (on which successive dolls rest) approaches the top rim. In other words, the sum of base thicknesses must converge to a total height equal to h, the height of the outer cup. S = h.

So if S = b / (1-r) and S = h then

h = b / (1-r)

...which is exactly what was already worked out above by considering just two dolls.

@LostInParadise: *In order for this to hold, what can we conclude about the relative heights of the tops and bottoms of each doll?* Do you mean the relative heights of the bottom and its base? The doll tops, it would seem, are an irrelevant red herring. Since they need not have any appreciable thickness, their proportions are not constrained as with the bottoms, so long as they decrease in size for nesting.