I don't think that interpretation of history is quite so strong, for a few reasons. While buying enough slaves to run a plantation was extremely expensive - buying a single slave to have for household purposes was not unobtainable or rare at all. Since slave status was inherited and the "one-drop" rule ensured it tended to get inherited, it was extremely common for less-financially well-off slave owners to simply rape an enslaved woman (multiple times over her lifetime) and then to either sell his mixed children, or commonly - to gift them as slaves to his white children when they come of age. This is a hugely valuable financial advantage for those who inherit a slave, and having a slave made them more valuable as a spouse as well, meaning they could increase their wealth more by marrying better than they could have without a slave. Nearly every white person from the south, has a family member who owned a slave - even if you would never guess by their economic status today. I'm from the south myself, and even though my great-grandparents were dirt poor in the mountains, I don't doubt someone somewhere back in time owned a slave. It really was just so common that the probability is a near certainty.
Also, when slavery was at it's height, Virginia, South Carolina, and Georgia were far more prosperous regions than they eventually became post-civil war. The south in general was more financially well-off during the time of slavery than it has been since - and you can read primary sources from the time and region saying as much. I can point you towards primary sources from elected officials of the time period saying as much - that they knew slavery was evil, but they also knew it was the source of economic and social stability that they enjoyed. The institution of slavery itself did not directly lead to economic fall of the southern states - rather it was the tensions from having a nation with both slaveholding states and non-slaveholding states, and the south's extremely strong incentive to protect their advantageous status even at the cost of war.
When the war did happen, the civil-war-related destruction that happened in the south (ie, the burning of Atlanta), had an equally dramatic impact and the outcome that we see today. Decades of slavery were certainly impactful, but so were the effects of the civil war, reconstruction, the great migration, and later cycles of industrialization, which tended to reach the south last due to network effects (there was no impetus to branch out from the skilled workers available up north... until unions took hold at least). All of those effects are more recent, and more relevant to the economic conditions today.
Additionally, the cultural regressiveness you speak of is closely tied to the Great Awakening and the particularly conservative religious denominations that became regionally dominant - and notably, these denominations had, and still have strong pull on southern white and black communities alike. The south actually had a reputation for being quite a fun and "sinful" place before that. My own state, NC, founded the first public university in the world (UNC Chapel Hill) - a paradoxically very progressive institution - despite being founded by slavers during a period of widespread slavery.
So understanding inequality and culture that exists in South, both now and then, and today is a complex thing. I would strongly caution trying to reduce it down that completely.