Live data from Hacker News

British Airways: All flights cancelled amid IT crash

bbc.com

101–110 of 255 posts

Re: British Airways: All flights cancelled amid IT crash

#101

Earlier quoted context omitted.

Land costs in the South East of England may be too high, but that really doesn't apply to the whole of the UK...

They are very high in most urban areas where you are likely to have a dev centre.

I'd say there are several locations in the UK with quite low costs (definitely massively lower than SE England).

For example Glasgow. Average residential prices are 23% of London, that's a lot cheaper and there's a good level of skilled workforce there.

Re: British Airways: All flights cancelled amid IT crash

#102
post #67

By using IT outsourcing in India (Tata?), Poland, BA and its parent is revealing some important information: They are cutting corners to the point of risking airline operations. The question is, where else are they unnecessarily cutting corners? Are they cutting unnecessary corners on airplane maintenance? Other quality airlines have outsourced IT or part of their IT operations, but they are careful about choosing th…

Quantas, the airline of Australia, outsources its IT operations to IBM. They did not choose the cheapest alternative, but a reliable one. I believe IBM was responsible for the Australian census debacle in 2016. Hardly a ringing endorsement of their reliability, and not the only high-profile instance of them messing things up. "No one ever got fired for choosing IBM" though.

And the IBM Phoenix payroll system disaster in Canada.

But hey at least Canadian government managers/executives got performance bonuses.

Re: British Airways: All flights cancelled amid IT crash

#103
post #34

The main KPI for BA's IT function is the percentage of jobs they have moved "nearshore" (to Krakow). Not that I think this is the Krakow workers' fault, mind you; rather this (and the string of similar IT incidents recently at BA) seems to fit the pattern of upper management focusing on driving down costs to the exclusion of everything else.

The system that's down was built, and is currently supported by, Tata Consultancy Services with support from BA IT staff. Once the system was built, BA made the majority of the IT department redundant.

I picture executives thinking of IT like rolling a big stone once it's moving get rid of the pushers.

Re: British Airways: All flights cancelled amid IT crash

#104
post #67

Earlier quoted context omitted.

Quantas, the airline of Australia, outsources its IT operations to IBM. They did not choose the cheapest alternative, but a reliable one. I believe IBM was responsible for the Australian census debacle in 2016. Hardly a ringing endorsement of their reliability, and not the only high-profile instance of them messing things up. "No one ever got fired for choosing IBM" though.

IBM makes mistakes but IBM is massive. You expect some things to happen just by the sheer surface area they have. I'd trust IBM over random outsourced coding houses when it comes to IT Security.

IBM can mismanage projects and deliver subpar results and people will keep giving them millions of dollars for it "because"

The smaller shop have more reasons to try harder and deliver a better result. IBM will just hire the cheapest people they can find and put them under 5 levels of management and good luck with that!

Re: British Airways: All flights cancelled amid IT crash

#105
post #39

Earlier quoted context omitted.

> If it were a high margin business, perhaps things would be different. It's not. Most of the loss here is not from lost bookings; a lot of people who prefer BA will probably just wait and book once the systems are restored. It's going to be from secondary losses -- lawsuits, ding to the reputation and so on. These numbers are estimatable, even for low-margin businesses.

I agree, but I've been around this sort of thing and seen the post incident analysis, been in the discussions, etc. Despite the huge costs of these outages, they pale in comparison to the costs of a real HA solution for all "needed to fly" applications. To give you some idea, ITA Software was bought by Google for $700 million. They were some of the best and brightest minds in this space, on par with any Silicon Valle…

I thought that they successfully delivered a passenger scheduling system, but it failed to gain market traction. I don't know the details, but I'd imagine it would be a huge migration cost even if the new system was much better.

Re: British Airways: All flights cancelled amid IT crash

#106

Earlier quoted context omitted.

Is there any evidence that IT outsourcing has made matters worse? Or is it only worse if the provider is located offshore? Arguably, delegating IT operations to a proper IT company could improve security and reliability. We use a low cost offshore IT provider and we haven't had any security or reliability issues so far ( https://gsuite.google.com/ )

The key is incentives. An IT outsourcer's goal is generally to fulfill their contractual obligations (to the letter not the spirit) for as little money as possible. So if something like security or reliability isn't clearly stated in contracts (and it's extremely difficult to do that well) it is disregarded. An outsourcing company is not going to spend money it doesn't have to, simple as that. SAAS like gsuite is an…

I don't see why an IT outsourcing company would not have the greatest incentive to take security and reliability extremely seriously.

Is there anything worse for an IT service provider than being blamed for a massive IT outage at a global corporation? This is headline news.

And I don't see any difficult contractual issues at all. On the contrary. A massive outage, by definition, means that the contractual obligation is not being met.

Re: British Airways: All flights cancelled amid IT crash

#107
post #97

Earlier quoted context omitted.

The reason is simple: short term thinking driven by short term investors

Or maybe it's what is repeated here so often: don't let the perfect be the enemy of the good. Perhaps it really is cheaper to have a few outages than it is to built the sort of systems that never go down. Every "nine" you add gets more and more expensive. Or perhaps the systems are so complex that it's just impossible for them to be fully reliable, and this is just one of the costs of having the ability to fly anywhe…

Or maybe it's what is repeated here so often: don't let the perfect be the enemy of the good.

Is that what you tell hundreds of thousands of passengers sitting in a terminal with canceled flights and confused airline employees during one of the biggest holiday weekends?

If your credit card number gets stolen or worse, your identity, and it's up to you to get that straightened out, are you just going to shrug your shoulders and write that off as the inevitable impossibility of perfection?

Re: British Airways: All flights cancelled amid IT crash

#109
post #67

Earlier quoted context omitted.

Quantas, the airline of Australia, outsources its IT operations to IBM. They did not choose the cheapest alternative, but a reliable one. I believe IBM was responsible for the Australian census debacle in 2016. Hardly a ringing endorsement of their reliability, and not the only high-profile instance of them messing things up. "No one ever got fired for choosing IBM" though.

While I can't speak to the exact circumstances, IBM when it takes over operations often hires the IT staff of the firm it is taking over operations from and runs systems on multiple-year contracts. At any rate, I believe few would contest that IBM in general is a more reliable vendor than Tata, other Indian suppliers or say, offices in Poland. EDIT: Details on IBM and Australian census. http://www.abc.net.au/news/201…

It's pretty funny how in this thread IBM is conflated with decent quality IT outsourcing and Poland with poor quality, while IBM's outsourcing is done out of Poland (among other countries).

https://www-03.ibm.com/press/us/en/pressrelease/32469.wss

Re: British Airways: All flights cancelled amid IT crash

#110
post #84

Earlier quoted context omitted.

Yes, it's a cyclic trend in IT: Step 0. Well functioning and balanced company. Step 1. Why these engineers are so expensive? We can hire 'ten for one' in 'country A'. Step 2. Cut local IT budget twice, put goals to have *zero* local IT budget in 2 years. Outsource everything to 'country A'. Step 3. People are fired, everything is outsourced, operational knowledge accumulated over the last few years is lost. Step 4. W…

Or, sometimes: Step 5. We need to hire high quality local team that can take ownership over product. We are willing to pay a fortune. But we can't hire anyone local, because we did step 3 for too long or too often, and that knowledge is locally gone.

I thought Step 5 would be: lets hire an overpriced western big name consulting firm to fix our 'temporary' problem.
Post reply on HN