Quite a lot. Social capital. Culture. About a hundred other things.
A set of concepts popularized goes under the names of cycle of poverty, culture of poverty, and generational poverty. Simply being poor seems less predictive of outcomes than multiple generation being poor. If your parents, your grandparents, and your great grandparents were poor, it's very difficult to break out.
In this case, "culture" isn't a euphemism for laziness or something crass like that; for example, there's a certain body of knowledge that goes with knowing how to move into the middle class. If no one in your family has that, it's very, very hard. That goes for everything from knowing what's required to apply for college, to knowing how much you're expected to learn when, to having the background to know how to support your kids' schooling.
Intervention programs which provide that background are effective. Even something as simple as letting parents know to read to little kids, how to make sure kids do homework, what supports are available, etc. Beyond that, having a guidance counselor who can let you know what schools you're likely to get into, what you need to apply, and what is and isn't appropriate to have in an admissions essay. These things aren't rocket science, but if neither you nor anyone around you has done them, they're very, very hard.