Half of European flights delayed due to system failure • Hacker News Board

Re: Half of European flights delayed due to system failure

#61

Post by tyingq » Wed, Apr 04, 2018, 2:00 AM UTC

tvanantwerp wrote: post #56

I was at a conference in Northern Virginia some months ago and saw a presentation from the folks at Upside, a startup specializing in booking business travel. They described the legacy system which handles pretty much all booking in the US, a system called SABRE. They described it as an ancient 6-bit computer system in Texas with no modern API. Everything they do tech-wise is a modern wrapper around that system. So I…

You're talking about TPF[1]. Many smart people and organizations have tried and failed to build something that could match it, including a company Google paid $700 million to buy. I personally know of at least 10 failed attempts :)

Not sure where "6 bit" is coming from though, and you can use gcc/c++ now, not just assembler[2]. And it's in Tulsa, Oklahoma, not Texas. Sabre's HDQ is in Texas, the data center is not. The hardware is very modern and new Z series mainframes in big loosely coupled clusters.

Amadeus, Sabre's main competition, also still has TPF at the core.

There is one notable non TPF reservation system. http://www.navitaire.com/ Last I checked, it couldn't scale well enough to handle a large airline.

Both Sabre and Amadeus are replacing TPF, but one function at a time (shopping, fare engine, booking, check in, etc). And very slowly.

[1]https://en.m.wikipedia.org/wiki/Transaction_Processing_Facil...

[2]https://www.ibm.com/support/knowledgecenter/en/SSB23S_1.1.0....

Fwiw, TPF is basically a huge, distributed, and transactionally consistent nosql database. Most orgs still using it have extracted most of the business logic that was in it, out to Linux boxes that front it. Not for stability reasons, but faster time to market with new features. To date, attempts to replace the high contention and high transaction rate nosql type traffic haven't scaled well enough.

Just in the US, 2.5 million people fly each day. And the processes to sell, board, etc each passenger are lots of transactions each. It's a pretty big scale. I think it's fairly close to Amazon sales per day, but with more contention and sub transactions.

All Visa credit card transactions are also still on TPF.

Re: Half of European flights delayed due to system failure

#62

Post by tim333 » Wed, Apr 04, 2018, 2:22 AM UTC

Apparently the Enhanced Tactical Flow Management System (ETFMS) packed up.

>“ETFMS facilitates improvements in flight management from the pre-planning stage to the arrival of the flight. It maximises the updating of flight-related data and thus improves the real picture of a given flight, thereby contributing to the Gate-to-Gate Concept,” Eurocontrol explains on its website.

>The agency initially reported that contingency procedures were immediately put in place which reduced the capacity of the European network by around 10 per cent. http://www.airtrafficmanagement.net/2018/04/eurocontrol-give...

They don't seem to say what went wrong but do say

>In over 20 years of operation, the ETFMS has only had one other outage which occurred in 2001. The system currently manages up to 36,000 flights a day.

Tech details from wikipedia fr:

Written in ADA , and running on HP-UX , the system is based on an exchange of messages between the airlines (who will file / change / update flight plans), the air traffic control bodies and the CFMU , the messages are written in ADEXP format.

ETFMS uses at least 5 fundamental notions:

flight plan : describes the 4D trajectory of an airplane. regulation: aircraft rate applied to a "volume traffic". Example: 50 aircraft / hour "Traffic volume": association of a geographical reference (air sector, waypoint, airport, etc.) and a set of aircraft flows. list of takeoff slots or called slot . Example: if the rate of regulation is 30 planes / h, there will be a slot every 2 minutes: 10 hours, 10 h 2, etc. the delay: difference in time between the take-off time desired by the company and the schedule calculated by ETFMS.

Re: Half of European flights delayed due to system failure

#63

Post by tyingq » Wed, Apr 04, 2018, 2:24 AM UTC

userbinator wrote: post #58

tvanantwerp wrote: post #56

I was at a conference in Northern Virginia some months ago and saw a presentation from the folks at Upside, a startup specializing in booking business travel. They described the legacy system which handles pretty much all booking in the US, a system called SABRE. They described it as an ancient 6-bit computer system in Texas with no modern API. Everything they do tech-wise is a modern wrapper around that system. So I…

I've heard of enough misguided "modernisations" (and failures thereof) that I think the "legacy system" was the part that stayed working throughout, and it's the newer stuff added around it that failed. The old stuff may be old but there's a reason it's old... it's outlasted any attempts at replacing it.

I can confirm this. It's usually the Java/Tomcat boxes that front the TPF box that cause this type of huge meltdown. It's almost never the TPF core. It happens, of course, but they've offloaded most of the functionality to more modern technology. So there isn't much code change in the TPF core. And code change is usually what drives outages.

Update: Seems this incident wasn't the reservation system at all. It was ETFMS. Government run system. http://www.eurocontrol.int/articles/enhanced-tactical-flow-m...

Re: Half of European flights delayed due to system failure

#64

Post by Pxtl » Wed, Apr 04, 2018, 2:30 AM UTC

jfktrey wrote: post #50

reificator wrote: post #49

Earlier quoted context omitted.

> If they'd use Docker with React this would have never happened! It's probably because they wrote it in Go. Did you know Go doesn't even have generics?

What do you mean? Go has generics. Kind of. https://www.reddit.com/r/rust/comments/5penft/parallelizing_...

Wait, is that string using inuktitut characters as fake angle-brackets? That is monstrously evil.

Re: Half of European flights delayed due to system failure

#65

Post by AceyMan » Wed, Apr 04, 2018, 2:30 AM UTC

userbinator wrote: post #58

tvanantwerp wrote: post #56

I was at a conference in Northern Virginia some months ago and saw a presentation from the folks at Upside, a startup specializing in booking business travel. They described the legacy system which handles pretty much all booking in the US, a system called SABRE. They described it as an ancient 6-bit computer system in Texas with no modern API. Everything they do tech-wise is a modern wrapper around that system. So I…

I've heard of enough misguided "modernisations" (and failures thereof) that I think the "legacy system" was the part that stayed working throughout, and it's the newer stuff added around it that failed. The old stuff may be old but there's a reason it's old... it's outlasted any attempts at replacing it.

>it's outlasted any attempt to replace it

See, https://en.wikipedia.org/wiki/Lindy_effect

(my fave citation from everyone's fave source for non-academic citations)

Re: Half of European flights delayed due to system failure

#66

Post by matte_black » Wed, Apr 04, 2018, 2:34 AM UTC

kylegordon wrote: post #41

matte_black wrote: post #36

Earlier quoted context omitted.

I would be 100% covered up to $500. I’d rather deal with a credit card company than a government anyway.

Your 'Merkin is showing. Wrong again. You deal with the airline. Not the 'government' If the airline does you wrong, at that point the CAA takes it up on your behalf. CAA ~= FAA

I have no desire to deal with an airline that has already fucked up a flight, and is probably already swamped. As soon as they give me the options for the next flight, I’ll take my own taxi, get my own hotel, and have Chase pick up the tab.

Re: Half of European flights delayed due to system failure

#67

Post by nkurz » Wed, Apr 04, 2018, 2:37 AM UTC

Pxtl wrote: post #64

jfktrey wrote: post #50

Earlier quoted context omitted.

What do you mean? Go has generics. Kind of. https://www.reddit.com/r/rust/comments/5penft/parallelizing_...

Wait, is that string using inuktitut characters as fake angle-brackets? That is monstrously evil.

Yup, that's even evil enough to quote in full:

  [–] pftbest 114 points 1 year ago 
  can you please explain this go syntax to me?
  type ImmutableTreeListᐸElementTᐳ struct {
  I thought go doesn't have generics.

  [–]Uncaffeinated[S] 239 points 1 year ago 
  It doesn't. That's just a "template" file, 
  which I use search and replace in order to
  generate the three monomorphized go files.
  If you look closely, those aren't angle brackets,
  they're characters from the Canadian Aboriginal
  Syllabics block, which are allowed in Go identifiers.
  From Go's perspective, that's just one long identifier.

https://www.reddit.com/r/rust/comments/5penft/parallelizing_...

Re: Half of European flights delayed due to system failure

#68

Post by matte_black » Wed, Apr 04, 2018, 2:38 AM UTC

greglindahl wrote: post #47

matte_black wrote: post #34

Earlier quoted context omitted.

The rental car insurance benefit already covered $2600 worth of damage when a rental car I had got banged up a bit. I also get a $300 travel credit a year.

Your normal car insurance covers rental cars.

This was another country.

Re: Half of European flights delayed due to system failure

#69

Post by makmanalp » Wed, Apr 04, 2018, 2:46 AM UTC

tim333 wrote: post #62

Apparently the Enhanced Tactical Flow Management System (ETFMS) packed up. >“ETFMS facilitates improvements in flight management from the pre-planning stage to the arrival of the flight. It maximises the updating of flight-related data and thus improves the real picture of a given flight, thereby contributing to the Gate-to-Gate Concept,” Eurocontrol explains on its website. >The agency initially reported that contin…

It would be fascinating to read about what bug took down a system with one outage in 20 years. I remember reading in the chubby paper (from google) that user error was the cause of more than half the downtime incidents. Wonder if that's the case here too.

Re: Half of European flights delayed due to system failure

#70

Post by tim333 » Wed, Apr 04, 2018, 2:55 AM UTC

makmanalp wrote: post #69

tim333 wrote: post #62

Apparently the Enhanced Tactical Flow Management System (ETFMS) packed up. >“ETFMS facilitates improvements in flight management from the pre-planning stage to the arrival of the flight. It maximises the updating of flight-related data and thus improves the real picture of a given flight, thereby contributing to the Gate-to-Gate Concept,” Eurocontrol explains on its website. >The agency initially reported that contin…

It would be fascinating to read about what bug took down a system with one outage in 20 years. I remember reading in the chubby paper (from google) that user error was the cause of more than half the downtime incidents. Wonder if that's the case here too.

Yeah, I recall last year

"BA’s £150,000,000 outage was caused by someone turning computers on and off too quickly"

I think companies often try to stay quite about the dumb stuff.