In the movie 'Its a Wonderful Life', why was the Bailey Building and Loan doing business with Henry Potter's bank?

George Bailey’s slightly inept uncle let the eight thousand dollars get misplaced in a newspaper to show Mr. Potter, when he went to deposit the money in—Potter’s bank?

If Mr. Potter wanted to shut down the Building and Loan for over 20 years, why didn’t he just refuse to do business with them? Wouldn’t that be a conflict of interest? And why was Henry Potter on the board of the Building and Loan? Early in the movie, he was an antagonist against Mr. Bailey’s father while on the board, and made a motion to shut down the Building and Loan down once Mr. Bailey (George Bailey’s father) died.