Is it controlling if you expect your partner home by a certain time?

So my boyfriend rarely ever goes out with the guys. He prefers to just have a calm night like relaxing with a beer talking to buddies, poker, fishing, movies, car meets/shows than drinking to get drunk and beer games like some of his other friends.

So he rarely goes out but when he does it’s usually really late. Late like past 4am and it really really bothers me. I don’t question his faithfulness to me but I absolutely hate it when he comes home near the time the sun comes up. I feel like coming home past 3am after the bars close is much.

Right now it’s 3am and he is still fishing with his friend. At first I wanted him to give me a time because he never knows when he’s going to be home and it would drive me crazy. So I give about 1:30am or 2am until I check in and he just says he will let me know when he leaves. But 3am comes and he’s not in bed when I wake and I start worrying. So now he’s like. “Just sleep. I’ll text you. I guess I’ll be home by 4:30am.”

Is it wrong for me to expect him home at a certain time? I feel like it’s wrong but I have no reasonable explanation to give. How would you deal without sounding like a mother?