Contractor underbid job - what is the protocol to try to recover some $?

My contractor gave me a bid on a job – a general fence installation that he has done “tons of times.” I told him my budget and the bid was within it and I also would give a 10–15% leeway for extra/unforseen costs. The total bid was $2000 ($1200 for labor, $800 for materials). The material total ended up being $1200 which is an extra 50% of the original quote!

I budgeted for this with some wiggle room but this seems way off, especially for someone “so experienced.” He has also been very hard to reach and a dingbat in general, making this whole process a mini hell.
Should I try to recover some of my costs, or is this just the way things work with Joe Blow contractor?