After a ground out, who does the first baseman throw to?

When a batter is thrown out at first on a ground ball, what determines the pattern of the "throw-around" after the play?

For example, it seems 1B will usually throw to SS, who throws to 3B, who throws back to the pitcher. Sometimes skipping the 2B. Does the 1B throw the ball back to the infielder who threw out the runner?