British Airways: All flights cancelled amid IT crash • Hacker News Board

Re: British Airways: All flights cancelled amid IT crash

#221

Post by justinclift » Sun, May 28, 2017, 5:23 AM UTC

yourapostasy wrote: post #214

nasalgoat wrote: post #206

Earlier quoted context omitted.

I receive about 10-20 unsolicited sales emails a day that are "personalized", which I promptly delete. If I am looking for a product, I will look for it. As far as I'm concerned, your emails are spam.

> I am looking for a product, I will look for it. Note that this attitude is what pushes some sales teams away from reaching out to line management, and towards C-level-targeted sales. If you hate "golf-cart sales" outcomes where a C-level forces a solution down your throat, then learning how to communicate with sales people will richly pay off. It is part of managing upwards, by short-circuiting sales warm approache…

In an ideal world, spammers wouldn't exist. Or at least C-level people who respond non-negatively, thus encouraging them to keep on doing it. :(

If people wonder why spam is a problem, it's pretty much because enough idiots respond to keep it worthwhile.

Re: British Airways: All flights cancelled amid IT crash

#222

Post by gaius » Sun, May 28, 2017, 6:15 AM UTC

sdfhbdf wrote: post #200

gaius wrote: post #142

Earlier quoted context omitted.

Polish engineers are sometimes a bit late when it comes to the new technologies, but are very good compared to ones from, for example, Ireland. It doesn't matter. They are no smarter than BA's original British engineers, but with decades less experience on those systems. Even if they were smarter - and in general the talent prefers to work for real companies, not bodyshops, so that's unlikely - they can't compensate…

> They are no smarter than BA's original British engineers, but with decades less experience on those systems Wow you really believe Poland is some third world country where we ride horses to our farms? They have experience, in some ways Poland is way ahead of UK in digitalization of many services. In great part because of thriving economy, great schools and excellent developers. Poland has some of the best programme…

Wow you really believe Poland is some third world country where we ride horses to our farms?

No, I was in Poland just last week in fact.

But my point stands: BA's own staff have been operating its systems for decades. No matter how good you are solving made-up puzzles on Hackerrank, you can't match that experience of those systems overnight. There would have been people at BA who'd been working on say the reservation system for 30+ years - and you're claiming your "hacker rank" can beat that. And THAT is why outsourcing always fails, that kind of belief.

Apologies if that seems a bit harsh, but the assertion that "we do well on Hackerrank therefore we are as qualified to maintain legacy systems as their original authors" just doesn't make sense.

Re: British Airways: All flights cancelled amid IT crash

#223

Post by gaius » Sun, May 28, 2017, 6:16 AM UTC

funkyy wrote: post #159

gaius wrote: post #142

Earlier quoted context omitted.

Polish engineers are sometimes a bit late when it comes to the new technologies, but are very good compared to ones from, for example, Ireland. It doesn't matter. They are no smarter than BA's original British engineers, but with decades less experience on those systems. Even if they were smarter - and in general the talent prefers to work for real companies, not bodyshops, so that's unlikely - they can't compensate…

Decades? We are talking here about 2-3 years lag due to conservatist approach. You seem to know nothing about what you write.

Decades? We are talking here about 2-3 years lag due to conservatist approach. You seem to know nothing about what you write.

You think the core systems of major airlines or any other large org turnover every 2-3 years? You think knowing the syntax of a particular language is comparable to decades of domain knowledge? You think that ANY 20-something, anywhere in the world, has decades of experience of anything??

No, YOU know nothing about what you write.

Re: British Airways: All flights cancelled amid IT crash

#224

Post by jl6 » Sun, May 28, 2017, 6:30 AM UTC

cyphar wrote: post #217

StavrosK wrote: post #179

Earlier quoted context omitted.

I've found that not only is it good to not assign blame in postmortems, but it's also accurate: The culprit usually is the checks and balances, as mistakes will happen, and the goal should be to have failsafes and detection. I'm reminded of airplane accidents: Whenever you hear of an airplane accident, it's always some amazingly crazy series of things going exactly wrong to get the plane to crash. We have a tendency…

> but a better way to think about it is that airplanes are so safe that an accident' can't occur unless a whole series of things go very specifically wrong. As an aside, I met someone who was working on a graph theory problem as their research project, and the application was that you could model the entire process of aircraft control through a state machine using that graph. Effectively they are working on making it…

> assuming that a certain process is followed

The challenge is to avoid pushing all the risk into that assumption. It's easy enough to build a system that never breaks if you're willing to assume perfect behaviour on the part of its dependencies, environment, users and operators.

Re: British Airways: All flights cancelled amid IT crash

#225

Post by tonyedgecombe » Sun, May 28, 2017, 7:27 AM UTC

rconti wrote: post #213

BillinghamJ wrote: post #174

Earlier quoted context omitted.

Their system should be built in a way which makes it resistant to a power supply issue. That is the fault of however built it, whether onshore or off.

Yeah. I'm sure the "power supply issue" is an overly simplistic explanation -- it's highly unlikely a single computer PS could cause such major issues in such a huge organization. That said, having almost entirely dodged any outsourcing-related issues in the 90s, and worked with generally great offshore teams, seeing my current role impacted by an utterly shortsighted and ignorant attempt to offshore critical operati…

I was worked on a site where a UPS failure took out a large AS400, the business stopped for three days while they waited for IBM to replace it.

Re: British Airways: All flights cancelled amid IT crash

#226

Post by GordonS » Sun, May 28, 2017, 9:21 AM UTC

One of the big take-aways from this is actually about how not to handle situations like this.

Problems happen - even huge ones - and I think most customes understand that. What they don't understand is being given little or no information about what they should do, and also being given vague, contradicting and even false information.

The BA Twitter feed seems to be the main source of dubious information. They are telling people to check the website for flight status info - but it only works intermittently, and different parts of the site say different things; for example, the flight status tool says my flight is cancelled, but the booking management tool says it's all fine.

On the Twitter feed they are telling people that they are contacting people and they rebooking them automatically - but it's apparent this is happening for very few people.

They are telling people to rebook on the website - which only works intermittently, and will not allow rebooking even when it is working.

They are telling people to call them to rebook - but their call centres seemed to be working on normal working hours (rather than getting all hand on deck), so were not in operation between 20:00 and 09:00 (or whatever; it varies by country). During working hours, when calling any call centre anywhere in the world, you just get a recorded message and are then disconnected. Some people say they did get into the call queue, and have been waiting on hold for 7+ hours!

They are sometimes telling people they shouldn't go to go to the airport to rebook, and sometimes telling them they should* go to the airport to rebook.

Yesterday they were telling people they could book alternative travel with a different airline and then claim it back from them... and today they are saying they won't pay if you book alternative travel - this could cost passengers dearly.

The CEO, Alex Cruz, also made a laughing stock of himself yesterday when he randomly donned a yellow high-visibility vest to do a recorded message in an office.

Honestly, the whole thing has been a lesson on how not to treat your customers when things go wrong.

Re: British Airways: All flights cancelled amid IT crash

#227

Post by sgt101 » Sun, May 28, 2017, 10:25 AM UTC

CommanderData wrote: post #167

I've managed enterprise IT applications shift support to India. I have to say when its done right, it can work well sometimes. But my experience has taught me this. The vast majority of time its a fragile egg shell waiting to crack. When it fails, it fails miraculously. An IT support team on this scale is one of those things you should keep close to your product or customer. Occasions like this I guarantee a bunch of…

But within 4 months no one in the BA C-suite will remember this happened.

Re: British Airways: All flights cancelled amid IT crash

#228

Post by cyphar » Sun, May 28, 2017, 10:35 AM UTC

jl6 wrote: post #224

cyphar wrote: post #217

Earlier quoted context omitted.

> but a better way to think about it is that airplanes are so safe that an accident' can't occur unless a whole series of things go very specifically wrong. As an aside, I met someone who was working on a graph theory problem as their research project, and the application was that you could model the entire process of aircraft control through a state machine using that graph. Effectively they are working on making it…

> assuming that a certain process is followed The challenge is to avoid pushing all the risk into that assumption. It's easy enough to build a system that never breaks if you're willing to assume perfect behaviour on the part of its dependencies, environment, users and operators.

If you've seen how aircraft controllers and pilots work, I think that "following the rules" is a very fair assumption to make. But ignoring that, obviously if implemented there would be fail-safes.

> It's easy enough to build a system that never breaks if you're willing to assume perfect behaviour

It isn't though. Seriously, think about how you could safely route several thousand flying hunks of metal through fairly small air corridors (which all have inertia) and you need to maintain strict flight schedules. Then think about how you need to factor in all of the edge cases caused by emergencies on planes (these are all included in the process for flight controllers). Then think how you could mathematically prove that safety.

Yes, it's easier if you assume that people will follow a certain process (and actual flight systems have so many layers of fail-safes that it's ridiculous) but it's definitely not "easy enough".

Re: British Airways: All flights cancelled amid IT crash

#229

Post by kbob » Sun, May 28, 2017, 11:18 AM UTC

tim333 wrote: post #58

Quite a lot of mixed gossip in the Daily Mail article >Yesterday's issues are the fourth BA failure in the past month, with problems on June 19, July 7 and July 13 >Union leaders say hundreds of BA staff complained about 'FLY' system and most workers say it's not fit for purpose >A survey by GMB of 700 staff in June found that 89 per cent said training was poor, 94 cent suffered delays or system failures and 76 per c…

That article is from July, 2016, which is why the dates "in the past month" don't compute.

Re: British Airways: All flights cancelled amid IT crash

#230

Post by gaius » Sun, May 28, 2017, 12:29 PM UTC

tyingq wrote: post #41

tomschlick wrote: post #31

Airlines should band together and form working group to redevelop the old system in modern open source tech. Yes it would probably take 5+ years to develop and roll out, but they can't keep maintaining these 50 year old mainframes that cost them tens of millions a year in downtime.

For what it's worth, it's not generally the mainframes that cause the outages. It's the distributed ecosystem where you need a bunch of disparate systems from different vendors around the mainframe, all coordinating, to be able to fly planes, sell tickets, process boarding passes, etc. There were fewer global outages when all of the functionality was on the TPF mainframes with dumb green screen clients.

TPF mainframes with dumb green screen clients

Implemented and operated by in-house staff.