How many children is too many?

I have 4 boys and would love to have another in the next year or so, I feel that it would complete my family and also would be my second child with my partner (3 from a previous relationship). However recently my little brother told me it was about time i stopped having kids and concentrated on the ones i already have! Now I look after all my children, pay for them, provide them with a loving stable home and treat them all the same. I adore my children’s company and would love to have one more. I don’t see i am doing anything wrong, but if all my family feels this way, should i be concerned? Am I going too far by having a 5th?