The media narrative is that there's a massive software failure going on in its scheduling system, and they need to stop flying for a day or two to "reset" it. Anyone have first-hand experience on their systems about what that really means?
Ask HN: What's Happening at Southwest?
1–10 of 106 posts
Re: Ask HN: What's Happening at Southwest?
#2Via reddit/r/bestof https://old.reddit.com/r/flying/comments/zw5lsl/southwest_pi...
"So the storm came and it impacted ground ops so bad that many many crews were now “unaccounted” for and the system in place couldn’t keep up. Then it happened for several more days. By Xmas evening the CS department had essentially reached the inability to do anything but simple, one off assignments. And to make matters worse, the phone system was updated not too long ago and it was not working well."
"I used to work for a large company trying to fill the void [huge gap in the market for good aviation scheduling software], and our software was damn good too. SW was one of the airlines interested, we would demo it exactly like the scenario today, but it was "too expensive" and they stayed on their homebuilt stuff."
Re: Ask HN: What's Happening at Southwest?
#3A key point is how they do routing - https://www.ajot.com/news/southwest-air-faces-gridlock-with-...
> Unlike competitors that use a so-called hub-and-spoke system to funnel passengers to large airports, Southwest is focused on point-to-point service, flying the same aircraft — Boeing Co. 737s — on trips that may hopscotch around the US.
With a hub and spoke system, all the planes go from A to HUB and then from the HUB to somewhere. If the route A-HUB gets saturated, they can put more planes on that, and those planes can always be found at the HUB. This applies to crews too.
You'll have something that looks like this: https://www.airlineroutemaps.com/maps/Delta_Air_Lines/North_...
This comes at the cost of having oversupply at some spots and its harder to offer the "ideal" routes as everyone needs to transfer to another plane with a layover somewhere... and your baggage is more likely to get lost. There's a bit not to like as a passenger on such an airline unless it's a nice one leg route - but then who wants to fly to Detroit?
Southwest is different - they go from anywhere they want to anywhere they want with non-stop flights and picking the most lucrative routes they can. This lowers the effective cost per flight and are likely non-stop flights. Everything that a customer wants.
Southwest routes from 2001 https://www.flickr.com/photos/erussell1984/15863298679 - you can see the lack of hubs there.
This is done through some crafty constraint programming to try to make sure that all the capacity is where it needs to be when it needs to be there.
However, when the capacity hits "holidays - everything is at max", along with "big storm prevents flights from going to where they need to be for the next leg" this system breaks down and planes and crews are out of position or need to sleep. Their software was able to handle this constraint system when it was a smaller company with fewer routes - there were fewer constraints.
The "reset" is not "shut down the computers and start them back up" but rather "let all the crews get their required sleep and then go to the spot where they need to be in order to handle the load - not where they currently are (out of position)".
Hypothetically, this is solvable if you have enough compute... but that's a lot of compute that needs to be recomputed each time something changes (weather, crew gets sick, passenger load changes) and that ends up being impractical and expensive.
---
Related reading:
NYT - What Caused the Chaos at Southwest : https://www.nytimes.com/2022/12/28/travel/southwest-airlines...
WSJ - How Southwest Airlines Melted Down : https://www.wsj.com/articles/southwest-airlines-melting-down... --- https://news.ycombinator.com/item?id=34165791
Re: Ask HN: What's Happening at Southwest?
#4How could a CEO let this persist for more than an hour? Is there nobody with the leaderships skills to just give everyone free flights and promise everything will be sorted while having people record who worked and who flew and what services were used as best they can?
The software failed, just Jerry-rig something up to get people where they need to be, let the accountants and programmers clear up the mess after the fact.
[Edit]Software helps optimize things, but as long as the crews and suppliers have sufficient faith that they will be treated fairly, you could run this all with paper and pen for a few days while the software gets straightened out.
Decide which hubs are most important, and have someone work out what goes where by hand. Once those routes are up and running, you can work outward to the less trafficked airports. Just work on getting the most people to the places that help the most.
Crews know how many hours they've worked, and can track that themselves for the moment, or perhaps the Captain could do that. Everyone has cell phones, and could route around this damage.
The consensus here on HN seems to be to give in without trying, which I find disturbing.
Re: Ask HN: What's Happening at Southwest?
#5How could a CEO let this persist for more than an hour? Is there nobody with the leaderships skills to just give everyone free flights and promise everything will be sorted while having people record who worked and who flew and what services were used as best they can? The software failed, just Jerry-rig something up to get people where they need to be, let the accountants and programmers clear up the mess after the…
[deleted]
Re: Ask HN: What's Happening at Southwest?
#6How could a CEO let this persist for more than an hour? Is there nobody with the leaderships skills to just give everyone free flights and promise everything will be sorted while having people record who worked and who flew and what services were used as best they can? The software failed, just Jerry-rig something up to get people where they need to be, let the accountants and programmers clear up the mess after the…
Would that really work? How do you make sure too many people don’t show up to get on the same plane? How do you make sure every flight has a crew? How do you decide which flights to still run when only half the incoming planes actually showed up?
Re: Ask HN: What's Happening at Southwest?
#7Basically what I read was the former CEO was accountant educated and focused on stock price and financials and ignored system upgrades for like a decade. That coupled with their decentralized nature of operating (no hubs or star pattern) the recent storms caused their systems to not “know” where people or planes were even though the people knew where they were. That knowledge is the critical feature because hours of fly-time for pilots and airplanes is crucial for legally mandated rest periods, maintenance and logistics.
Having personally experienced the delays and issues with SW earlier this year I can suggest not flying them other than direct flights. Them not having any hubs means if one plane is late or a crew times out because of delays you are stranded. We got stranded at one airport overnight and got routed thru 2 other airports, changing planes at one, before reaching our destination. They have zero capacity of crews/planes on standby at any airport, probably because of the former CEO leadership and pandemic complications.
Re: Ask HN: What's Happening at Southwest?
#8A key point is how they do routing - https://www.ajot.com/news/southwest-air-faces-gridlock-with-... > Unlike competitors that use a so-called hub-and-spoke system to funnel passengers to large airports, Southwest is focused on point-to-point service, flying the same aircraft — Boeing Co. 737s — on trips that may hopscotch around the US. With a hub and spoke system, all the planes go from A to HUB and then from the H…
I find it hard to believe that compute power is the limitation here… something like simulated annealing would probably provide a reasonable, even if far from optimal solution on a regular PC. I imagine the feeding all of the data and constraints in is a bigger issue, when there is a lot of confusion and disorganization.
Re: Ask HN: What's Happening at Southwest?
#9Basically what I read was the former CEO was accountant educated and focused on stock price and financials and ignored system upgrades for like a decade. That coupled with their decentralized nature of operating (no hubs or star pattern) the recent storms caused their systems to not “know” where people or planes were even though the people knew where they were. That knowledge is the critical feature because hours of…
Ah so bean counting treated IT as a cost center and cut CapEx to the bone. Running things with no reserved capacity, crew wise and system wise, is just asking for a catastrophic fall off the cliff style shutdown.
Edit: a simple queue theory can explain why no reserved capacity leads to system breakdown.
Re: Ask HN: What's Happening at Southwest?
#10I don't think the reset is around resetting the software logic exactly, but rather that things have gotten so far outside of what they should be they no longer know where planes are, where crews that can find those planes are, and how many people even are still planning on using Southwest to fly to their intended destination.
They need a "reset" to spend a few days inputting as much of that data as possible into their computer system, at which point they can start operating with some level of efficiency and start functioning again.