Live data from Hacker News

British Airways: All flights cancelled amid IT crash

bbc.com

31–40 of 255 posts

Re: British Airways: All flights cancelled amid IT crash

#31
Airlines should band together and form working group to redevelop the old system in modern open source tech.

Yes it would probably take 5+ years to develop and roll out, but they can't keep maintaining these 50 year old mainframes that cost them tens of millions a year in downtime.

Re: British Airways: All flights cancelled amid IT crash

#32

The main KPI for BA's IT function is the percentage of jobs they have moved "nearshore" (to Krakow). Not that I think this is the Krakow workers' fault, mind you; rather this (and the string of similar IT incidents recently at BA) seems to fit the pattern of upper management focusing on driving down costs to the exclusion of everything else.

Krakow is a big tech center, is in the EU, and it's not obvious why moving jobs there would have anything to do with a drop in quality.

It's not about Krakow per se. But when a company works towards specific targets like "90/10 supplier offshore ratio", rather than metrics based on quality or efficiency, I don't think quality comes first.

Re: British Airways: All flights cancelled amid IT crash

#33

One customer was told that the root cause was a "lightning strike on a datacentre", although it sounds pretty unlikely (there are storms today in the UK but surely there'd be a DR plan?)

Surely they have multiple datacentres as redundancy.

Re: British Airways: All flights cancelled amid IT crash

#34

The main KPI for BA's IT function is the percentage of jobs they have moved "nearshore" (to Krakow). Not that I think this is the Krakow workers' fault, mind you; rather this (and the string of similar IT incidents recently at BA) seems to fit the pattern of upper management focusing on driving down costs to the exclusion of everything else.

The system that's down was built, and is currently supported by, Tata Consultancy Services with support from BA IT staff. Once the system was built, BA made the majority of the IT department redundant.

Re: British Airways: All flights cancelled amid IT crash

#35

One customer was told that the root cause was a "lightning strike on a datacentre", although it sounds pretty unlikely (there are storms today in the UK but surely there'd be a DR plan?)

Surely they have multiple datacentres as redundancy.

The two data centres are geographically very close to each other, straddling the runway.

Re: British Airways: All flights cancelled amid IT crash

#36
post #35

Earlier quoted context omitted.

Surely they have multiple datacentres as redundancy.

The two data centres are geographically very close to each other, straddling the runway.

It's also not generally that easy. There's more than one system required to operate flights, usually more than 10. Some would be in BA data centers (UK), some in Amadeus/Altea data centers(DE), etc. Some applications are vendor provided, some in-house. So, you can have two of everything. But when you have something big, like a power outage, getting everything talking to the "surviving/dr" instances of all the different sytems isn't easy. And, the outage itself causes a huge spike in client driven transactions. Every passenger and employee is hitting whatever the equivalent of a reload button is.

The fix is more expensive than any major airline has been willing to spend thus far.

Re: British Airways: All flights cancelled amid IT crash

#37
post #25

One customer was told that the root cause was a "lightning strike on a datacentre", although it sounds pretty unlikely (there are storms today in the UK but surely there'd be a DR plan?)

They do have DR plans, but the reality is that whole thing is a huge distributed system with parts from different vendors, in different locations, etc, with decades of legacy. A power outage the reboots a whole datacenter would screw any major airline for at least half a day. Thus far, nobody has found the economics of a modern, truly HA setup worth the cost. Outsiders greatly underestimate the complexity too. Think…

> If it were a high margin business, perhaps things would be different. It's not.

Most of the loss here is not from lost bookings; a lot of people who prefer BA will probably just wait and book once the systems are restored.

It's going to be from secondary losses -- lawsuits, ding to the reputation and so on.

These numbers are estimatable, even for low-margin businesses.

Re: British Airways: All flights cancelled amid IT crash

#38

Earlier quoted context omitted.

Krakow is a big tech center, is in the EU, and it's not obvious why moving jobs there would have anything to do with a drop in quality.

It's not about Krakow per se. But when a company works towards specific targets like "90/10 supplier offshore ratio", rather than metrics based on quality or efficiency, I don't think quality comes first.

I agree with you on that. But I had to stick up for Krakow!

Re: British Airways: All flights cancelled amid IT crash

#39
post #25

Earlier quoted context omitted.

They do have DR plans, but the reality is that whole thing is a huge distributed system with parts from different vendors, in different locations, etc, with decades of legacy. A power outage the reboots a whole datacenter would screw any major airline for at least half a day. Thus far, nobody has found the economics of a modern, truly HA setup worth the cost. Outsiders greatly underestimate the complexity too. Think…

> If it were a high margin business, perhaps things would be different. It's not. Most of the loss here is not from lost bookings; a lot of people who prefer BA will probably just wait and book once the systems are restored. It's going to be from secondary losses -- lawsuits, ding to the reputation and so on. These numbers are estimatable, even for low-margin businesses.

I agree, but I've been around this sort of thing and seen the post incident analysis, been in the discussions, etc. Despite the huge costs of these outages, they pale in comparison to the costs of a real HA solution for all "needed to fly" applications.

To give you some idea, ITA Software was bought by Google for $700 million. They were some of the best and brightest minds in this space, on par with any Silicon Valley darling. They successfully wrote a modern replacement for one popular airline function...shopping. They failed, however, at delivering a modern reservation system, despite tons of money and talent invested.

Re: British Airways: All flights cancelled amid IT crash

#40

Airlines should band together and form working group to redevelop the old system in modern open source tech. Yes it would probably take 5+ years to develop and roll out, but they can't keep maintaining these 50 year old mainframes that cost them tens of millions a year in downtime.

"Redevelop the old system in modern open source tech" is a tough sell when so many of these airlines are differentiating on technology. Myself and many other frequent fliers will not engage with companies with poor functionality and bad UX. We move or stay loyal when airline tech allows us to do & see most of the things we need.
Post reply on HN