Do you pay a mechanic if he's unable to fix or even diagnose the problem?

Depends.. If he tells you you need to pay for a estimate. Then you should pay.

If they say free estimate then you shouldn’t.

Pay for the estimate, but (obviously) not for them fixing the problem.

You need to get clear on the terms before having them start any kind of estimate or work. I’d suggest expressing that it is a question to you, and if they are going to charge you for time to estimate, be sure you express any limits you have on how much you are willing to pay before you’d want them to stop.