You outline the problem very well. The banking sector has an ENORMOUS amount of power over the economy/society in a way that no other industry does because it literally has the ability to create money out of thin air (by loaning it into existence). The banking industry is the arbiter and credit in our society and decides where money get allocated.
This would be fine if it were their own money, but it's not - it's our bank accounts (ie. fractional reserve banking). Unfortunately individuals can't opt out of this because unlike private banks, individuals aren't allowed to open a bank account directly with the government (though these replies show that this is possible in countries like France).
The finance industry does not generate real wealth, it is a wealth extracting industry that profits off the spread between the interest rate decided by the central bank and the interest rates and fees it charges consumers. Banks do not create goods or services, they just decide who gets the money to do so, with money that's not theirs. 80% of bank credit goes to mortgage loans (driving up housing prices and saddling homeowners in debt). Banks have gone from 2% of the U.S. economy in the 1950s to 8% by 2008, and 1.5% of the British economy in 1978 to 15% by 2008.
The first step to fixing this is to give citizens the ability to opt out of private banks and bank directly with the central bank. Private banks should not be the only ones with this privilege.