What can I do to help my hand recover from an injury that happened 4 years ago?

I punched a brick wall while wearing thick winter gloves and broke the left pinky and wrist four years ago. It hurts when I lift more than a large milk jug. I went to the doctor and he said if it gets painful I can get a x-ray, but other than that he can’t help me. Its called a boxers injury and is very common.