Japanese maple tree is not dropping dead leaves?

My Japanese maple tree in the backyard is about 5 feet tall with a similar size of canopy. The leaves are dead now but still on the tree. It was the same last winter but I had left them on the tree until they finally dropped. Should I remove the leaves by hands? It is unsightly.