Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

British Airways: All flights cancelled amid IT crash

bbc.com

Post reply on HN
Print view View original

131–140 of 255 posts

Re: British Airways: All flights cancelled amid IT crash

#131

Post by gaius »

mancerayder wrote: post #63

In the tech-related (but now fully mainstream, unlike a few years ago) news these days, there are a number of recurring themes. 1. Massive security breaches affecting major corporations, governments, and so on. 2. Massive IT outages, affecting corporations, governments and so forth. It doesn't take a lot of incidents to mark them as 'massive' since things are strongly centralized in this world. Meanwhile, in the last…

that CTO's would want to spend more and not LESS on operational costs

The incentives are the same as they always were. A well run IT organization can coast on momentum for a long time - everything is neatly automated. So they sack their good engineers and hire an outsourcing company and save an amazing amount of money just long enough to get that next bonus or promotion, and they're long gone when the momentum runs out and the wheels come off. Repeating the same scam at another company.

Re: British Airways: All flights cancelled amid IT crash

#132

Post by tudorconstantin »

raesene9 wrote: post #96
fauigerzigerk wrote: post #86

Earlier quoted context omitted.

Is there any evidence that IT outsourcing has made matters worse? Or is it only worse if the provider is located offshore? Arguably, delegating IT operations to a proper IT company could improve security and reliability. We use a low cost offshore IT provider and we haven't had any security or reliability issues so far ( https://gsuite.google.com/ )

The key is incentives. An IT outsourcer's goal is generally to fulfill their contractual obligations (to the letter not the spirit) for as little money as possible. So if something like security or reliability isn't clearly stated in contracts (and it's extremely difficult to do that well) it is disregarded. An outsourcing company is not going to spend money it doesn't have to, simple as that. SAAS like gsuite is an…

I've worked in outsourcing for 9 years in Romanian companies as a software developer. I had the impression that the goal was to have the customer satisfied, not to fulfill a contract (as long as the customer was also reasonable, of course) and for pragmatic reasons: a happy customer pays you longer. My longest project was for 4 years, with the same customer. And my former employer keeps working on that project, 2 years after I left the company. In these 5 years the customer's company got acquired and for some reasons they had a pretty high turnover with their own employees, but they kept working with my employer.

When I joined the company there were 60 employees, when I left it 5 years later, there were about 500. I think they could only achieve this growth based on their high work ethics and the quality of their hiring their system.

Re: British Airways: All flights cancelled amid IT crash

#133

Post by lordnacho »

mancerayder wrote: post #63

In the tech-related (but now fully mainstream, unlike a few years ago) news these days, there are a number of recurring themes. 1. Massive security breaches affecting major corporations, governments, and so on. 2. Massive IT outages, affecting corporations, governments and so forth. It doesn't take a lot of incidents to mark them as 'massive' since things are strongly centralized in this world. Meanwhile, in the last…

I liken it to shorting options. Let me explain.

In the derivatives world (that I started my career in) there's a common theme: you can sell options, and generally you will make money. The only issue is, now and again you lose a lot of money. Often enough to wipe out everything you made.

But incentives matter. If you come in as a new guy, you want to impress by making money. And come up with some story about how you'll know when not to sell options. You also don't want to be the guy losing money when the other desks are making it. If there's a crisis and everyone loses, that's fine. Unforeseen, right? It happened to everyone.

The CTO is facing a very similar choice. Save money by not upgrading, work the security guys a bit harder and let them update the OS a bit slower. Don't think too hard about redundancy. Or code review, that takes away dev time. You're saving money.

But now and again, something happens. Maybe you have to leave, but you'll still have "saved money" for several years before. You'll land somewhere.

Re: British Airways: All flights cancelled amid IT crash

#134

Post by ivan_gammel »

fauigerzigerk wrote: post #106
raesene9 wrote: post #96

Earlier quoted context omitted.

The key is incentives. An IT outsourcer's goal is generally to fulfill their contractual obligations (to the letter not the spirit) for as little money as possible. So if something like security or reliability isn't clearly stated in contracts (and it's extremely difficult to do that well) it is disregarded. An outsourcing company is not going to spend money it doesn't have to, simple as that. SAAS like gsuite is an…

I don't see why an IT outsourcing company would not have the greatest incentive to take security and reliability extremely seriously. Is there anything worse for an IT service provider than being blamed for a massive IT outage at a global corporation? This is headline news. And I don't see any difficult contractual issues at all. On the contrary. A massive outage, by definition, means that the contractual obligation…

If the possible negative consequences to company's image is the only incentive, they will rather try find an excuse, since they've fulfilled the requirements of the contract. Can be anything, from missing security requirements (under the assumption that their customer will take care about security himself) to incorrect use of provided software etc. And they'll probably be right, especially if they actually tried to sell the security. And they'll probably be able to defend their position in court and sue for defamation.

Re: British Airways: All flights cancelled amid IT crash

#135

Post by gaius »

mbaha wrote: post #64
benmarks wrote: post #40

Earlier quoted context omitted.

"Redevelop the old system in modern open source tech" is a tough sell when so many of these airlines are differentiating on technology. Myself and many other frequent fliers will not engage with companies with poor functionality and bad UX. We move or stay loyal when airline tech allows us to do & see most of the things we need.

Their tech is not open enough to even allow for proper innovation and differentiation. That's the main issue with legacy IT. When you live in a world w/ Docker, REST (and every other open tech there is), you can build systems which are way more innovative.

When you live in a world w/ Docker, REST (and every other open tech there is), you can build systems which are way more innovative

That's quite funny because Docker and REST are just half-arsed reinventions of mainframe features from the 1970s.

Re: British Airways: All flights cancelled amid IT crash

#136

Post by ams6110 »

mancerayder wrote: post #107
ams6110 wrote: post #97

Earlier quoted context omitted.

Or maybe it's what is repeated here so often: don't let the perfect be the enemy of the good. Perhaps it really is cheaper to have a few outages than it is to built the sort of systems that never go down. Every "nine" you add gets more and more expensive. Or perhaps the systems are so complex that it's just impossible for them to be fully reliable, and this is just one of the costs of having the ability to fly anywhe…

Or maybe it's what is repeated here so often: don't let the perfect be the enemy of the good. Is that what you tell hundreds of thousands of passengers sitting in a terminal with canceled flights and confused airline employees during one of the biggest holiday weekends? If your credit card number gets stolen or worse, your identity, and it's up to you to get that straightened out, are you just going to shrug your sho…

I'm saying yes, maybe that is the case. Maybe those scenarios are either a) cheaper to handle on occasion than building systems which will never fail, or b) unavoidable (with some low probability).

Air travel itself is not 100% safe. Planes crash. People die. Yet we feel that the probabilities are enough in our favor that we accept it.

Re: British Airways: All flights cancelled amid IT crash

#137

Post by gonzo »

mtkd wrote: post #16
tyingq wrote: post #9

Earlier quoted context omitted.

Sort of. The software is called "automated re-accommodation". But you're solving for several intertwined problems. Which aircraft models / tail numbers fly which flights...they have different seating capacities, nautical range, etc. Which crews are assigned to which aircraft. They aren't all qualified to fly every model. And, they aren't all in the right city, so you have to "deadhead" them there. And, finally, which…

I was in the middle of some major rail cancellations a couple of months back over 2 days - on the 3rd day many of the assets were out of place around the country - they were clearly trying to solve it manually and the impact escalated over the day even though most of the original problems had cleared it's got to be a great problem to work on - and must be pretty rewarding to watch when you get it right

Remember on 9/11/2001 when the US FAA declared a nation-wide ground stop, and rerouted all inbound international flights to other countries (e.g. Canada https://en.wikipedia.org/wiki/Operation_Yellow_Ribbon)?

Re: British Airways: All flights cancelled amid IT crash

#138

Post by gaius »

riffic wrote: post #89
al452 wrote: post #82

Earlier quoted context omitted.

The UK government (perhaps the relevant one in this case) does not have a stellar track record of figuring out how to make big IT systems and projects successful. This knee-jerk appeal to regulation is unconvincing to say the least.

A modified form of ITIL would do a lot of good for important sectors. Let me put it another way, if the industry won't regulate itself, then they will be regulated by force.

A modified form of ITIL would do a lot of good for important sectors

You've never used ITIL for real, have you? Because if you had you would know that no amount of process can replace good engineers, and good engineers don't want to work in ITIL shops...

Re: British Airways: All flights cancelled amid IT crash

#139

Post by fauigerzigerk »

sgt101 wrote: post #119
fauigerzigerk wrote: post #106

Earlier quoted context omitted.

I don't see why an IT outsourcing company would not have the greatest incentive to take security and reliability extremely seriously. Is there anything worse for an IT service provider than being blamed for a massive IT outage at a global corporation? This is headline news. And I don't see any difficult contractual issues at all. On the contrary. A massive outage, by definition, means that the contractual obligation…

Here is the list of things that are worse for an IT outsourcer. 1. Being more expensive than the other guy.

I doubt that anyone wants to compete solely on price. It's a margins game.

Re: British Airways: All flights cancelled amid IT crash

#140

Post by pbhjpbhj »

>BA chief executive Alex Cruz said: "We believe the root cause was a power supply issue." //

That shut down BA operations World wide? Does that seem likely? Possible? They don't have power fail-over and operational centres in different countries?

I've been re-shaping my aluminium foil hat and wondering if there wasn't a specific terrorist threat that's been covered up; but then where I am there have been 2 cities suffer bomb threats (with attendance of bomb disposal and armed police) that seem to have been buried in the news completely. Also I've heard elsewhere that staff on the ground reported the incident as due to "hackers" almost immediately - the speculation being 'before they could possibly have known that' - which suggests some sort of disinformation process.

/wild-speculation

Post reply on HN