How much should you weight up the other parent's mental health and insist visits be limited or postponed, for the welfare of the child?

I keep hearing reports from other family members that my son’s father is very ill, not eating, very depressed, talking in a confusing way, refusing to go to work or deal with bills. He was ranting and crying that I was keeping my son away. (which isn’t true, he over-nighted two nights in a row a few weeks ago when he was feeling up.) He doesn’t spend that much time with our son. He doesn’t call and when he wants to see his son on a weekend (which isn’t every weekend…), he sends an sms on the day or really late on a Friday. We have a written agreement that he is to phone on the Monday to plan for the following weekend, but he has never done this. He never talks in the way a regular parent does, like ‘Would little man like to come hang with me this weekend?’ He says, ‘I have to see my son.’ or ‘I want to see him this weekend.’ I think my ex is using our son to cheer himself up, but I don’t know how healthy for my son some of the visits have been when he is really down. Sorry this is so long, but it’s been a bit rough lately.