Why should, in the U.S., churches be exempt from taxes?

I can understand tax deductions for churches that run food banks, Salvation Army shelters and things like that but why should church property be exempt from real estate taxes? Why should the clergy be exempt from social security taxes? Why should donations to church building funds and missionary work be tax exempt? Just because the exemptions span religions (even Scientology) it still seems like taxing non-members and government support of religion.