Some reasons which are cited in the literature, of which there exists lots (
http://www.kansascityfed.org/publicat/research/community/Unb... is a good entry point):
1) Many poor people mistrust banks, for reasons ranging from cultural ("Where I come from, they're far more likely to vanish with your cash than the mattress") to personal experience ("I was just going about my business one day but then I ran the account a little low, $300 vanished in fees, my rent check bounced and I got arrested for 'uttering.'").
2) Banks have some assumptions made about their customers which are institutionalized by practice and sometimes regulation or policy. These assumptions map fairly well to the middle class but sometimes don't travel well. Here's four things I could tell you about an adult member of the middle class: they're literate, have a primary address which changes relatively infrequently, carry identification with them, and do not fear speaking to Authority because they're not cognizant of a reason why that should result in their immediate arrest.
3) "Free checking" was a multi-billion dollar ongoing wealth transfer from poor bank customers to banks and relatively wealthier bank customers. ("Free checking" was a marketing tactic very popular in the 90s and early 2000s where banks eliminated monthly fees for checking accounts while slashing minimum balances to open them. The flipside is that they then started cranking up fees generated by e.g. overdrafts. This combination generally tends to not be to the benefit of (many) poor people.)
4) The alternative financial sector has a product mix which more closely matches poor people's needs than traditional banks. For example, short-term loans for small amounts of money available with minimal documentation required are a) astoundingly high risk, b) relatively expensive to process, c) have fairly low margins, and d) carry explosive reputational risk. These counsel against the typical retail bank even trying to offer them. Much better to loan a middle class American $8k out of a credit card limit of $20k at 12% with a 3% loss rate then to loan a poor person $100 with $110 due a week later. (If you APR-ize that it becomes "A number which one never, ever wants to get asked about when you're called to testify before Congress about how you're causing poverty in the community you serve.") Business thus flows to the alternative financial sector.
5) (The literature is a bit circumspect about this one, but it's there.) Sometimes poor people make decisions which, if you appreciated their limited options, individualized viewpoints of the world, and the sociocultural context and structural issues at play... are still very effing bad decisions.