Am I being fair in not letting my children have a relationship with their deadbeat father?

My boys don’t know their bio-dad (The last time my 4 yr old saw him he called him “That Man”. Ive tried in the past to let him into their lives but everytime he will see them once or twice and then disappear for several months when he gets back with his girlfriend (according to him his GF will not let him see them!! lol ) They already know someone else as “dad” and I dont want them confused or hurt. Him and his girlfriend broke up again and he contacted me wanting to see them after almost a year. I told him I would think about it. Im not being malicious in not wanting him to see them. If he were a responsible man that truly loved his children I would be more than happy to have him be a part of their lives. What is the right path to take?!?