Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's Happening at Southwest?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 106 posts

Re: Ask HN: What's Happening at Southwest?

#11

Post by burkaman »

mikewarot wrote: post #4

How could a CEO let this persist for more than an hour? Is there nobody with the leaderships skills to just give everyone free flights and promise everything will be sorted while having people record who worked and who flew and what services were used as best they can? The software failed, just Jerry-rig something up to get people where they need to be, let the accountants and programmers clear up the mess after the…

You can't give people free flights if you don't know where your planes and crews are and don't know where to tell your angry customers to go. There are also laws to adhere to, there is a maximum amount of time a flight crew can work so if you tell them to "just fly if you can and we'll figure it out later" you're probably going to break a lot of laws and endanger a lot of passengers.

I think they probably are trying to jerry-rig a system, but the airline industry is heavily regulated for safety reasons (a good idea that has been extremely successful), so it's very difficult to get a plane in the air if you don't know what the fuck you're doing.

Re: Ask HN: What's Happening at Southwest?

#12

Post by shagie »

_yb2s wrote: post #8
shagie wrote: post #3

A key point is how they do routing - https://www.ajot.com/news/southwest-air-faces-gridlock-with-... > Unlike competitors that use a so-called hub-and-spoke system to funnel passengers to large airports, Southwest is focused on point-to-point service, flying the same aircraft — Boeing Co. 737s — on trips that may hopscotch around the US. With a hub and spoke system, all the planes go from A to HUB and then from the H…

I find it hard to believe that compute power is the limitation here… something like simulated annealing would probably provide a reasonable, even if far from optimal solution on a regular PC. I imagine the feeding all of the data and constraints in is a bigger issue, when there is a lot of confusion and disorganization.

It would, and that's likely what they're using - but it isn't perfect.

When it becomes non-perfect and you get a number of events that throw it off (like storms causing certain legs from not getting completed to move the plane to the proper spot and holidays causing disproportionate load in certain parts of the network), then everything gets messy.

Re: Ask HN: What's Happening at Southwest?

#13

Post by hrunt »

mtmail wrote: post #2

Via reddit/r/bestof https://old.reddit.com/r/flying/comments/zw5lsl/southwest_pi... "So the storm came and it impacted ground ops so bad that many many crews were now “unaccounted” for and the system in place couldn’t keep up. Then it happened for several more days. By Xmas evening the CS department had essentially reached the inability to do anything but simple, one off assignments. And to make matters worse, the ph…

> "I used to work for a large company trying to fill the void [huge gap in the market for good aviation scheduling software], and our software was damn good too. SW was one of the airlines interested, we would demo it exactly like the scenario today, but it was "too expensive" and they stayed on their homebuilt stuff."

This is a common situation in any industry. A company has a solution it self-developed in its infancy to support itself, and as it grows, it comes to rely very heavily on this software. At a certain point, everyone involved with the software -- from the developers to the users to the customers -- knows that it is not on par with third-party software developed by a team dedicated to all of its potential use-cases and pitfalls, but it is very difficult for the business to replace it because the ongoing cost is a relatively low maintenance cost, and the replacement cost is a one-time, relatively high purchase fee (with, usually, a similar if not higher ongoing maintenance cost). So the immediate reaction is to stay with the low ongoing cost, even though everyone "knows" that the long-term benefits of the third-party solution far outweigh the financial savings of keeping the in-house solution, whether those benefits might be avoiding a national week-long service outage or something simpler, like the ability for staff to get more done in other areas of the company that need attention when getting larger.

I have been there. It can be very difficult to make the financial case when the future benefits are somewhat speculative or intangible. Accountants tend to devalue those benefits, relative to the hard numbers they already have. It does not help that system replacement projects often go over budget and introduce other problems that cost more money. Replacing home-grown systems is hard because they are not just software replacements; they typically also require reworking business processes.

Re: Ask HN: What's Happening at Southwest?

#14

Post by bombcar »

shagie wrote: post #3

A key point is how they do routing - https://www.ajot.com/news/southwest-air-faces-gridlock-with-... > Unlike competitors that use a so-called hub-and-spoke system to funnel passengers to large airports, Southwest is focused on point-to-point service, flying the same aircraft — Boeing Co. 737s — on trips that may hopscotch around the US. With a hub and spoke system, all the planes go from A to HUB and then from the H…

I wonder how long the “direct chains” are - if a plane is used on a flight on Monday how long until it runs that leg again, if ever?

Re: Ask HN: What's Happening at Southwest?

#15

Post by phil21 »

From what I know... Not affiliated at all with the airline industry, but I fly a lot and tend to nerd out on this stuff...

Their crew (and flight) scheduling software functions in a manner where it more or less simulates a "perfect" day of operations. Airplanes take off on time, land, and continue on. If anything disrupts this simulation, crew members had to call in and talk to someone to update the computer system to tell it that both the airplane and crew members are not where the system thought they were.

Once the call center got overwhelmed it was a cascading failure with Southwest quickly not understanding where most of their flight crew happened to be at any given moment. It appears they feel the only way to solve this is get everything (planes and crews) back into "starting position" to restart the simulation.

Re: Ask HN: What's Happening at Southwest?

#16

Post by rjh29 »

mikewarot wrote: post #4

How could a CEO let this persist for more than an hour? Is there nobody with the leaderships skills to just give everyone free flights and promise everything will be sorted while having people record who worked and who flew and what services were used as best they can? The software failed, just Jerry-rig something up to get people where they need to be, let the accountants and programmers clear up the mess after the…

It's so simple, why didn't this entire company full of experienced specialists just do the random idea I just came up with without thinking about it at all?

Re: Ask HN: What's Happening at Southwest?

#17

Post by Simon_O_Rourke »

phil21 wrote: post #15

From what I know... Not affiliated at all with the airline industry, but I fly a lot and tend to nerd out on this stuff... Their crew (and flight) scheduling software functions in a manner where it more or less simulates a "perfect" day of operations. Airplanes take off on time, land, and continue on. If anything disrupts this simulation, crew members had to call in and talk to someone to update the computer system t…

In this age of more or less fully network connected devices, this kind of a setup seems archaic in the extreme, probably likely to be some hangover from the 70s or 80s.

Re: Ask HN: What's Happening at Southwest?

#18

Post by mark-r »

hrunt wrote: post #13
mtmail wrote: post #2

Via reddit/r/bestof https://old.reddit.com/r/flying/comments/zw5lsl/southwest_pi... "So the storm came and it impacted ground ops so bad that many many crews were now “unaccounted” for and the system in place couldn’t keep up. Then it happened for several more days. By Xmas evening the CS department had essentially reached the inability to do anything but simple, one off assignments. And to make matters worse, the ph…

> "I used to work for a large company trying to fill the void [huge gap in the market for good aviation scheduling software], and our software was damn good too. SW was one of the airlines interested, we would demo it exactly like the scenario today, but it was "too expensive" and they stayed on their homebuilt stuff." This is a common situation in any industry. A company has a solution it self-developed in its infan…

The existing solution needn't even be home built. I worked for a company that replaced one ERP system with another. It was a medical manufacturer, so in many cases the new system had to warp to the established government mandated business practices. Over time and over budget, you bet - they weren't finished with the transition when I left.

Re: Ask HN: What's Happening at Southwest?

#19

Post by gitfan86 »

From what I've read here and other places there was a unique situation of:

1. Large holiday volume of flights AND a lot of cancelations due to weather happened at the same time. 2. Their software system was unable to recommend optimal next steps for flight crews because it had not been designed to take this specific situation into account.

Other airlines also had problem #1. But since they mostly operate on a hub model it is easy to tell their flight crews where they should go once the weather gets better.

Re: Ask HN: What's Happening at Southwest?

#20

Post by _yb2s »

shagie wrote: post #12
_yb2s wrote: post #8

Earlier quoted context omitted.

I find it hard to believe that compute power is the limitation here… something like simulated annealing would probably provide a reasonable, even if far from optimal solution on a regular PC. I imagine the feeding all of the data and constraints in is a bigger issue, when there is a lot of confusion and disorganization.

It would, and that's likely what they're using - but it isn't perfect . When it becomes non-perfect and you get a number of events that throw it off (like storms causing certain legs from not getting completed to move the plane to the proper spot and holidays causing disproportionate load in certain parts of the network), then everything gets messy.

It doesn’t seem like that, they have canceled virtually all flights to and from airports unaffected by storms. They seem to have huge numbers of planes and flight crews ready to go that are just sitting idle.
Post reply on HN