1. Moving stuff in a ledger from A to B (sending money around in general) is all fairly trivial from a technology standpoint, but only a minuscule part of what financial institutions are doing. They spend a lot of money and effort on combatting fraud and making sure rules and regulations (KYC, AML, CTF, sanctions, etc.) are being followed.
(The crypto bros (and some FinTechs) solve a tiny part of the puzzle with a new technology (not even good/efficient tech in case of crypto), and then think that they alone have solved it, and that the rest is easy, and that they are now in a position to take over finance. Not so.)
2. Costs vary tremendously by jurisdiction and industry structure. In Europe, bank transfers within the SEPA region are basically free (instantaneous transfers might cost 35 cents or so). Bank transfers, direct debit, and standing orders have been ubiquitous and cheap for a long time, and checks and credit cards are rarely used (on the continent; the UK is closer to the US system). There was no need for PayPal, as it was fairly trivial to pay bills. (Only when Ebay bought PayPal and made it the default payment method did PayPal get any traction in Europe, I think.)
Similarly, credit card interchange fees are capped at 0.5% in the EU. That means fewer card rewards (which arguably benefit mostly the rich), and less credit card marketing.
The banking system in the US is quite crap, but that's not due to stupidity, but a mix of a) well intended regulation that intends to support small regional banks, b) misguided business-friendly regulation with insufficient consumer protection.
2.b. I think Visa/MC/Amex are ridiculously expensive in many jurisdictions, and that they basically skim off some 2% or so of the entire retail revenue is ridiculous. They should be treated as cheap infrastructure. But this requires sensible regulation. It's not a technological issue.
3. There are lots of somewhat successful neo-banks, neo-brokers, etc. in many jurisdictions. I don't think many of them have extraordinary profits. And that's because, no, it's not "all coming from inefficiency and lack of real competition". Sure, the sector is so heavily regulated (like aviation) that you can't just walk in and compete willy-nilly. That has some downsides, but it also has advantages.
4. A bank that makes insufficient profits and tethers on the brink of insolvency invites bank runs. That's one reason regulators are not pushing too hard for more competition. (Replacing the current fractional banking system with private credit + narrow banking might be an option, but that's a huge and complicated topic...)
For deep insight, I recommend Bits about Money by Patrick McKenzie (patio 11 on HN).
https://www.bitsaboutmoney.com
https://news.ycombinator.com/user?id=patio11