Why do doctors who work in a group think it is ok to say a patient cannot switch to another doctor within the group?

I know this does not happen in every group, but I have heard of it more than once.

Another pet peeve of mine is calling for an appointment, either me or my primary care doctor, stating the medical problem, and being told every doctor in the group handles the problem, when in fact later to find out certain doctors in the group actually specialize. For instance this happened to me with a neurology referral; I wound up with a doctor who generally works with children, and I am 43 year old woman. I now am having a problem for a very specific very rare problem, and want to make sure I see someone who has a reasonable amount of experience before driving over two hours to a specialist and paying out a bunch of money. Why can’t I get a simple answer? My doctors office is calling with the referral, my doctor recommended I see a specialist, this is not me just wanting to direct my care.

So back to main question, I have a fear of being sent to the wrong doctor, and then he not being willing to give me up as a patient to a doctor in that office who might be able to better help me.