Should I convert some of my term life insurance to whole life?

If I understand it all correctly the whole life will earn 5% and I can cancel the policy and get my money back, and the interest is not taxed until I pull it out? Is that correct? Or, do I have to wait 30 years to be able to early withdraw the money? Basically, it is like a taxed deferred savings plan? Or, is the whole amount taxed if you withdraw the money or collect the money upon death? Then the money paid in is taxed twice.

What are the postives and negatives in your opinion? One obvious negative is the plan costs a lot compared to term.