Why is celibacy in the Roman Catholic priesthood a concern of anyone except those who take the vow?

In a recent question regarding the new pope, it was mentioned that it is a shame he follows the line of the church regarding celibacy of the clergy. Why is this choice criticised? What impact does it have on anyone other than the clergy themselves?