How do I grow a new herb plant from a branch of an existing larger plant?

There's a huge rosemary bush in my backyard and I want to grow a new plant. I cut a branch and have it sitting in a jar of water hoping new roots will sprout, but some how this doesn't seem like enough. Suggestions?