How can I politely get the annoying neighbor to stop coming over?

Every time my husband and I are out in the yard, our goofy/annoying neighbor comes over. Our dog doesn’t like him either. Despite that our dog can’t stand him, the neighbor likes to peek over the fence and talk to him. (which makes our dog bark like crazy) With the dog barking now, the neighbor now wants to stand in our driveway and talk about absolutely nothing. Sometimes we are busy, but he just doesn’t get the hint. He isn’t retarded, but he’s probably got the mental capacity of a 10 year old. I feel bad for him, but doesn’t he have something better to do? I will now spend the next 10 minutes trying to get the dog to stop barking.