Live data from Hacker News

United Airlines System-Wide Computer Problem

flyertalk.com

111–117 of 117 posts

Re: United Airlines System-Wide Computer Problem

#111

I worked on United's computer systems for a year (never that one though), and so I get nervous when I see a headline like that. True story: one of their systems still runs on a mainframe that has 9 bits in a byte!

Sabre, still runs all their reservations through ztpf... If you know anyone there ask them about how they get the data off the mainframe. The story I heard sounds like the process was designed by Rube Goldberg.

Re: United Airlines System-Wide Computer Problem

#112
post #96

Nice reminder about a "glitch" happening in one of the datacenters use at the airline I used to work. Went like this: Guy who shows around the new datacenter/ops guy demonstrated how the emergency power off works by lifting the protection plate. Protection plate unhinges suddenly and droppes onto emergency power off button. Hilarity ensues.

A data center where I work had a naked big red switch without a protection plate. Until I flagged up this lack and let the IT people know about it, it was a disaster waiting to happen. There's a plate on the switch now, praise Eris.

I've seen many places where "the solution" to the unprotected EPO button was just a simple plastic cup!

Re: United Airlines System-Wide Computer Problem

#113
post #59
post #17

United is a mess. I had the misfortune of flying with them a couple months ago. I ended up in a city a 2 hour drive from my actual destination and had to rent a car on my own to get to where I was going. That was the worst of it, but almost every flight I saw on the way (both ones I was on and other flights at nearby gates) was delayed or overbooked or otherwise messed up in some way.

If I ever listened to horror stories like these, there wouldn't be an airline left I would fly with. This is textbook; replace United with Delta, Southwest, Air Canada, etc., at will. The only company I've used that I haven't heard exactly this type of complaint about is Widerøe, a tiny regional line in Norway with a fleet of prop planes. This is unfortunate, as I live in Pennsylvania and get motion sickness on those…

It's also worth remembering that quality of service depends on a host of variables, including departure/arrival cities (major hubs are better than regional airports), routes, and times (morning flights tend to be less delayed, Thursday afternoon flying always a bit of a shitshow), etc.

I fly out of UA hubs frequently and have had nothing but excellent service from them this year (over 50 segments flown this year, ~60K miles).

Definitely helps to have status too...

Personally, I consider SW to the be shittiest of them all. I hate having to fight for a seat...

Lastly, use google.com/flights by far my favorite booking tool now.

Re: United Airlines System-Wide Computer Problem

#114

Earlier quoted context omitted.

So obviously the "move fast and break things" philosophy is meant for some fun web applications. But what's the equivalent "modern best practice" for systems that are much more weighed towards stability and resilience as opposed to new features?

Systemic quality focus would be a good start. Deming-based management philosophy driving a systems-oriented model. This actually applies to all businesses; speed improves, market fit improves (quality is just "what is good and valuable" after all), employee happiness improves (exponential gains from that alone). The effects are systemically positive. But, no one cares, and the belief that we have to crack the whip to…

Well, on one hand there's six sigma and then there's the agile manifesto.

That's quite a wide gulf.

Would love to hear from people who have worked with developing / managing relatively high quality software with relatively modern methods.

Edit: Maybe NASA's Faster Better Cheaper comes to mind...

Re: United Airlines System-Wide Computer Problem

#115

Earlier quoted context omitted.

Systemic quality focus would be a good start. Deming-based management philosophy driving a systems-oriented model. This actually applies to all businesses; speed improves, market fit improves (quality is just "what is good and valuable" after all), employee happiness improves (exponential gains from that alone). The effects are systemically positive. But, no one cares, and the belief that we have to crack the whip to…

Well, on one hand there's six sigma and then there's the agile manifesto. That's quite a wide gulf. Would love to hear from people who have worked with developing / managing relatively high quality software with relatively modern methods. Edit: Maybe NASA's Faster Better Cheaper comes to mind...

Six Sigma and Agile are indeed at pretty opposite sides of the spectrum.

Deming -- W. Edwards Deming, that is -- is somewhere in the middle, around the right balance. He advocated for spreading a philosophy of systems thinking, scientific method, and statistical understanding, while simultaneously empowering employees by recognizing the power and responsibility of management and leadership, and understanding the motivation from a scientifically accurate psychological viewpoint.

It's a correct framework, and it's all aimed at driving quality by improving the things that directly impact it at a base level: fundamentally, how people work together, how they build systems that work, and how they're motivated (and demotivated) in reality.

Re: United Airlines System-Wide Computer Problem

#116

Earlier quoted context omitted.

You might be right, unfortunately. If it's cheaper to buy insurance that will cover the (expected) losses caused by outages, most organizations will choose to do that instead of making the software more failure-resistant. The problem is that insurance only works well for isolated incidents, but a software failure can cause a cascading failure with a huge impact. Insurance companies generally aren't prepared for that…

But aren't the insurance companies smart enough to figure this out and start correcting their rates to be much higher?And if they actually have their acts together, wouldn't those same insurance companies start insisting on basic audits of their client's systems? I actually don't know about this stuff, so any correction of my thoughts is appreciated.

Quite often operational losses aren't insured against this kind of error, for example the Knight-Ridder automated trading losses. Sufficiently big operational failures can just destroy companies, especially small companies.

System audits would have to be standardised. There's bits of this in ISO9001, PCI compliance, FIPS, and so on. But the technology changes rapidly and the insurance companies don't have the expertise.

Re: United Airlines System-Wide Computer Problem

#117
post #89
post #55

Earlier quoted context omitted.

Truth is that most reservation systems are built on TPF and it isn't really easy to replace.

just in case we have readers to whom TPF isn't a household name: https://en.wikipedia.org/wiki/Transaction_Processing_Facilit...

That sounds like a heck of a system.
Post reply on HN