British Airways: All flights cancelled amid IT crash • Hacker News Board

Re: British Airways: All flights cancelled amid IT crash

#41

Post by tyingq » Sat, May 27, 2017, 3:52 PM UTC

tomschlick wrote: post #31

Airlines should band together and form working group to redevelop the old system in modern open source tech. Yes it would probably take 5+ years to develop and roll out, but they can't keep maintaining these 50 year old mainframes that cost them tens of millions a year in downtime.

For what it's worth, it's not generally the mainframes that cause the outages. It's the distributed ecosystem where you need a bunch of disparate systems from different vendors around the mainframe, all coordinating, to be able to fly planes, sell tickets, process boarding passes, etc.

There were fewer global outages when all of the functionality was on the TPF mainframes with dumb green screen clients.

Re: British Airways: All flights cancelled amid IT crash

#42

Post by theoh » Sat, May 27, 2017, 4:14 PM UTC

tyingq wrote: post #9

mtkd wrote: post #7

Earlier quoted context omitted.

are there commonly algos or decision support tools to help unravel that in an optimised way?

Sort of. The software is called "automated re-accommodation". But you're solving for several intertwined problems. Which aircraft models / tail numbers fly which flights...they have different seating capacities, nautical range, etc. Which crews are assigned to which aircraft. They aren't all qualified to fly every model. And, they aren't all in the right city, so you have to "deadhead" them there. And, finally, which…

One vendor of those optimization tools is Jeppesen, which about ten years ago bought up two previously independent companies in the sector, SBS (New York) and Carmen Systems (in Gothenburg, Sweden).

Re: British Airways: All flights cancelled amid IT crash

#43

Post by branchless » Sat, May 27, 2017, 4:16 PM UTC

markonen wrote: post #10

The main KPI for BA's IT function is the percentage of jobs they have moved "nearshore" (to Krakow). Not that I think this is the Krakow workers' fault, mind you; rather this (and the string of similar IT incidents recently at BA) seems to fit the pattern of upper management focusing on driving down costs to the exclusion of everything else.

Land costs are too high in the UK. Everything except banking must leave.

Re: British Airways: All flights cancelled amid IT crash

#44

Post by jacques_chester » Sat, May 27, 2017, 4:16 PM UTC

tyingq wrote: post #39

jacques_chester wrote: post #37

Earlier quoted context omitted.

> If it were a high margin business, perhaps things would be different. It's not. Most of the loss here is not from lost bookings; a lot of people who prefer BA will probably just wait and book once the systems are restored. It's going to be from secondary losses -- lawsuits, ding to the reputation and so on. These numbers are estimatable, even for low-margin businesses.

I agree, but I've been around this sort of thing and seen the post incident analysis, been in the discussions, etc. Despite the huge costs of these outages, they pale in comparison to the costs of a real HA solution for all "needed to fly" applications. To give you some idea, ITA Software was bought by Google for $700 million. They were some of the best and brightest minds in this space, on par with any Silicon Valle…

> Despite the huge costs of these outages, they pale in comparison to the costs of a real HA solution for all "needed to fly" applications.

Then, quite honestly, it's the rational decision for airlines to take.

Even Google accepts that perfect reliability is impossible, and they're sailing in a pillow-strewn gold-plated yacht down a Mississippi of money.

Re: British Airways: All flights cancelled amid IT crash

#45

Post by blibble » Sat, May 27, 2017, 4:21 PM UTC

idlewords wrote: post #27

markonen wrote: post #10

The main KPI for BA's IT function is the percentage of jobs they have moved "nearshore" (to Krakow). Not that I think this is the Krakow workers' fault, mind you; rather this (and the string of similar IT incidents recently at BA) seems to fit the pattern of upper management focusing on driving down costs to the exclusion of everything else.

Krakow is a big tech center, is in the EU, and it's not obvious why moving jobs there would have anything to do with a drop in quality.

if the offshored personnel work via a bodyshop then the incentives are completely different compared to direct employees, and the resulting work is considerably lower quality (at least in my experience)

if offshored personnel are direct employees it's a completely different story

given the main cost saving comes from being able to "ramp the size of the pool up and down to meet demand" most large companies opt for the bodyshop solution...

Re: British Airways: All flights cancelled amid IT crash

#46

Post by abhianet » Sat, May 27, 2017, 4:28 PM UTC

tomschlick wrote: post #31

Airlines should band together and form working group to redevelop the old system in modern open source tech. Yes it would probably take 5+ years to develop and roll out, but they can't keep maintaining these 50 year old mainframes that cost them tens of millions a year in downtime.

Is that not how Amadeus happened?

Re: British Airways: All flights cancelled amid IT crash

#47

Post by gjjrfcbugxbhf » Sat, May 27, 2017, 4:36 PM UTC

benmarks wrote: post #40

tomschlick wrote: post #31

Airlines should band together and form working group to redevelop the old system in modern open source tech. Yes it would probably take 5+ years to develop and roll out, but they can't keep maintaining these 50 year old mainframes that cost them tens of millions a year in downtime.

"Redevelop the old system in modern open source tech" is a tough sell when so many of these airlines are differentiating on technology. Myself and many other frequent fliers will not engage with companies with poor functionality and bad UX. We move or stay loyal when airline tech allows us to do & see most of the things we need.

They can build a open-source back-end and differentiate in the front end.

Or even build common components/ technologies together and then build their back-ends on top - that is basically what the rest of us are doing...

Re: British Airways: All flights cancelled amid IT crash

#48

Post by [deleted] » Sat, May 27, 2017, 4:46 PM UTC

gjjrfcbugxbhf wrote: post #47

benmarks wrote: post #40

Earlier quoted context omitted.

"Redevelop the old system in modern open source tech" is a tough sell when so many of these airlines are differentiating on technology. Myself and many other frequent fliers will not engage with companies with poor functionality and bad UX. We move or stay loyal when airline tech allows us to do & see most of the things we need.

They can build a open-source back-end and differentiate in the front end. Or even build common components/ technologies together and then build their back-ends on top - that is basically what the rest of us are doing...

[deleted]

Re: British Airways: All flights cancelled amid IT crash

#49

Post by raesene9 » Sat, May 27, 2017, 4:52 PM UTC

branchless wrote: post #43

markonen wrote: post #10

The main KPI for BA's IT function is the percentage of jobs they have moved "nearshore" (to Krakow). Not that I think this is the Krakow workers' fault, mind you; rather this (and the string of similar IT incidents recently at BA) seems to fit the pattern of upper management focusing on driving down costs to the exclusion of everything else.

Land costs are too high in the UK. Everything except banking must leave.

Land costs in the South East of England may be too high, but that really doesn't apply to the whole of the UK...

Re: British Airways: All flights cancelled amid IT crash

#50

Post by coldcode » Sat, May 27, 2017, 4:52 PM UTC

Airlines these days are run by a complex set of systems most of which have to work in order to have the airline function. I remember a few years ago SABRE (one of the 3 major GDS companies in the world) had a 4 hour outage (I worked for a division). Half the world's airlines stopped functioning. Generally these things seldom go down but when they do, hell breaks loose. Often these systems not only do reservations, but crew scheduling, weight balancing, check-in, manifest creation and a whole host of other small but important things. Airlines also integrate some of their own pieces into the contracted ones, and some just contract almost everything. It usually comes down to money. Even Southwest Airlines which doesn't share its res data with anyone uses a GDS backend for a lot of things (in this case SABRE).