Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

British Airways: All flights cancelled amid IT crash

bbc.com

Post reply on HN
Print view View original

211–220 of 255 posts

Re: British Airways: All flights cancelled amid IT crash

#211

Post by madeofpalk »

davidf18 wrote: post #62

By using IT outsourcing in India (Tata?), Poland, BA and its parent is revealing some important information: They are cutting corners to the point of risking airline operations. The question is, where else are they unnecessarily cutting corners? Are they cutting unnecessary corners on airplane maintenance? Other quality airlines have outsourced IT or part of their IT operations, but they are careful about choosing th…

> Quantas, the airline of Australia, outsources its IT operations to IBM. They did not choose the cheapest alternative, but a reliable one.

Actually, Qantas has a huge 'workforce' of Tata onsite as well. Of course all the newer and 'more exciting' stuff is ran either by in-house resources or 'digital agencies' - the more expensive kind of outsourcing.

(FWIW, it's Qantas - Queensland and Northern Territory Aerial Services)

Re: British Airways: All flights cancelled amid IT crash

#212

Post by devonkim »

StavrosK wrote: post #179
2manyredirects wrote: post #158

One of the unions has been quick to attribute the issue to the outsourcing to India of some of the IT responsibility, which the right wing press here has been all too eager to publish, but BA have rebuffed this and said at this stage they believe the root cause was a power supply issue - sure, that could be attributed somewhere along the lines to an 'alpha male business asshole' (as I read in one of the comments here…

I've found that not only is it good to not assign blame in postmortems, but it's also accurate: The culprit usually is the checks and balances, as mistakes will happen, and the goal should be to have failsafes and detection. I'm reminded of airplane accidents: Whenever you hear of an airplane accident, it's always some amazingly crazy series of things going exactly wrong to get the plane to crash. We have a tendency…

One other important point is that the very term "root cause" is extremely harmful in that it presumes a primary failure and already seeds the idea of one bad actor and, by proxy, blame. Systems today are too complex to blame upon one or two things - we operate in a very complicated, "complected" world both in our software and in many organizations.

While there are always technical causes for larger technical failures, I've seen far too many times RCA post-mortems performed that result in witch hunts instead of a solemn contemplation of how things could be better done by everyone. Such an RCA may ignore that a normally careful engineer was overworked by managers, never is lack of relevant monitoring and testing due to budget cuts cited, and you'll certainly never see "teams X and Y collaborated too much" as a reason for failure in these places. Because in a typical workplace, the company's values and culture are never related to a failure. You can't objectively measure how bad or how good a culture is either. Why make it part of post mortems when you don't think it's a failure?

Re: British Airways: All flights cancelled amid IT crash

#213

Post by rconti »

BillinghamJ wrote: post #174
2manyredirects wrote: post #158

One of the unions has been quick to attribute the issue to the outsourcing to India of some of the IT responsibility, which the right wing press here has been all too eager to publish, but BA have rebuffed this and said at this stage they believe the root cause was a power supply issue - sure, that could be attributed somewhere along the lines to an 'alpha male business asshole' (as I read in one of the comments here…

Their system should be built in a way which makes it resistant to a power supply issue. That is the fault of however built it, whether onshore or off.

Yeah. I'm sure the "power supply issue" is an overly simplistic explanation -- it's highly unlikely a single computer PS could cause such major issues in such a huge organization.

That said, having almost entirely dodged any outsourcing-related issues in the 90s, and worked with generally great offshore teams, seeing my current role impacted by an utterly shortsighted and ignorant attempt to offshore critical operations tasks is quite disheartening. It's rarely the fault of the teams, it's the fault of higher-ups who completely fail to grasp the complexity and consequences of the tasks they're offloading. Everything looks great for a few weeks or months or years until one of the dozens of things that have gone neglected rear their ugly heads. If they're lucky, they take the money and run before the crash happens and escape most blame.

Re: British Airways: All flights cancelled amid IT crash

#214

Post by yourapostasy »

nasalgoat wrote: post #206
ajeet_dhaliwal wrote: post #194

Earlier quoted context omitted.

There is a distinction between spam email and a personalized targeted sales email. Mine was most definitely not spam.

I receive about 10-20 unsolicited sales emails a day that are "personalized", which I promptly delete. If I am looking for a product, I will look for it. As far as I'm concerned, your emails are spam.

> I am looking for a product, I will look for it.

Note that this attitude is what pushes some sales teams away from reaching out to line management, and towards C-level-targeted sales. If you hate "golf-cart sales" outcomes where a C-level forces a solution down your throat, then learning how to communicate with sales people will richly pay off. It is part of managing upwards, by short-circuiting sales warm approaches to the management you report to, and turning them into your allies to help you pitch your priorities that happen to align with their sales goals.

I am up front with the sales people who approach me, and tell them when I anticipate the problem space they solve will rotate onto my front burners, my anticipated budget to switch (usually in the form of "if I switch to what you propose, I only have $X to do it with, all in, software and services"), the benefits from my current solution I want to ensure stay in place, and the pain points I want to solve. That usually ends the conversation right there and then, I'm tagged "no-contact" in their CRM, and the spam disappears.

Re: British Airways: All flights cancelled amid IT crash

#215

Post by rconti »

fauigerzigerk wrote: post #86
mancerayder wrote: post #63

In the tech-related (but now fully mainstream, unlike a few years ago) news these days, there are a number of recurring themes. 1. Massive security breaches affecting major corporations, governments, and so on. 2. Massive IT outages, affecting corporations, governments and so forth. It doesn't take a lot of incidents to mark them as 'massive' since things are strongly centralized in this world. Meanwhile, in the last…

Is there any evidence that IT outsourcing has made matters worse? Or is it only worse if the provider is located offshore? Arguably, delegating IT operations to a proper IT company could improve security and reliability. We use a low cost offshore IT provider and we haven't had any security or reliability issues so far ( https://gsuite.google.com/ )

You're not wrong that delegating operations COULD improve security and reliability, but it could also suffer from the same issues that branch office suffer from -- out of sight, out of mind, distance from the corporate HQ culture and scrutiny, etc.

Furthermore, often these issues rear their ugly heads when they sack the existing staff who are keeping 1000 balls in the air and rarely dropping one. In the migration to an outsourced team, or an offshore team, or even just another equally competent team in the same office, details will get missed. It's simply not possible to do a complete knowledge transfer. Things will get missed, and sooner or later one of the things you miss will turn out to be a landmine.

Of course, the original team wasn't perfect either, and could have also suffered a large outage, but the new team is more likely to do so, at least in the short term.

In a best-case outsourcing scenario, you either hire better people, or more people, or a vendor with better processes or understanding. You do the migration very carefully and miss as few details as possible, AND the quality of the new team is such that, within 3 or 6 or 12 months, they're actually outperforming the original team. AND you manage to avoid hitting any landmines during the transition period where they're underperforming. This requires a great bit of planning, execution, and luck.

Re: British Airways: All flights cancelled amid IT crash

#216

Post by rconti »

pbhjpbhj wrote: post #140

>BA chief executive Alex Cruz said: "We believe the root cause was a power supply issue." // That shut down BA operations World wide? Does that seem likely? Possible? They don't have power fail-over and operational centres in different countries? I've been re-shaping my aluminium foil hat and wondering if there wasn't a specific terrorist threat that's been covered up; but then where I am there have been 2 cities suf…

Could have been a power feed to a datacenter. We can't always take newspaper descriptions as literally as we might like.

Re: British Airways: All flights cancelled amid IT crash

#217

Post by cyphar »

StavrosK wrote: post #179
2manyredirects wrote: post #158

One of the unions has been quick to attribute the issue to the outsourcing to India of some of the IT responsibility, which the right wing press here has been all too eager to publish, but BA have rebuffed this and said at this stage they believe the root cause was a power supply issue - sure, that could be attributed somewhere along the lines to an 'alpha male business asshole' (as I read in one of the comments here…

I've found that not only is it good to not assign blame in postmortems, but it's also accurate: The culprit usually is the checks and balances, as mistakes will happen, and the goal should be to have failsafes and detection. I'm reminded of airplane accidents: Whenever you hear of an airplane accident, it's always some amazingly crazy series of things going exactly wrong to get the plane to crash. We have a tendency…

> but a better way to think about it is that airplanes are so safe that an accident' can't occur unless a whole series of things go very specifically wrong.

As an aside, I met someone who was working on a graph theory problem as their research project, and the application was that you could model the entire process of aircraft control through a state machine using that graph. Effectively they are working on making it mathematically impossible for a crash to occur assuming that a certain process is followed (with safety measures ofc).

Re: British Airways: All flights cancelled amid IT crash

#218

Post by contingencies »

tomschlick wrote: post #31

Airlines should band together and form working group to redevelop the old system in modern open source tech. Yes it would probably take 5+ years to develop and roll out, but they can't keep maintaining these 50 year old mainframes that cost them tens of millions a year in downtime.

I'm willing to commit to a serious effort in this area if anyone wants to collaborate - email in profile. I've been digging in to real time logistics for my existing business recently anyway, previously founded a 3300+ hotel reservation network and contributed a lot of Wikipedia content around PNR records and their political background/intelligence use, so have some familiarity with industry structure and protocols, and built and defended HA 24x7 transaction infrastructure for a major Bitcoin exchange. I would seek people with more experience close to or inside the major booking networks or airlines as collaborators.

Re: British Airways: All flights cancelled amid IT crash

#219

Post by justinclift »

ajeet_dhaliwal wrote: post #194
justinclift wrote: post #189

Earlier quoted context omitted.

Well, it certainly sounds like you spammed them. :(

There is a distinction between spam email and a personalized targeted sales email. Mine was most definitely not spam.

Ahhh. That's a misconception on your part. There is literally no difference between any kind of sales email, and spam. All sales email to people that haven't opted-in previously, is spam. 100% of it. Thankfully, some countries have even introduced laws against it.

That's the reason for the FOAD response.

Re: British Airways: All flights cancelled amid IT crash

#220

Post by devonkim »

fauigerzigerk wrote: post #86
mancerayder wrote: post #63

In the tech-related (but now fully mainstream, unlike a few years ago) news these days, there are a number of recurring themes. 1. Massive security breaches affecting major corporations, governments, and so on. 2. Massive IT outages, affecting corporations, governments and so forth. It doesn't take a lot of incidents to mark them as 'massive' since things are strongly centralized in this world. Meanwhile, in the last…

Is there any evidence that IT outsourcing has made matters worse? Or is it only worse if the provider is located offshore? Arguably, delegating IT operations to a proper IT company could improve security and reliability. We use a low cost offshore IT provider and we haven't had any security or reliability issues so far ( https://gsuite.google.com/ )

There's some evidence in software engineering research that offshoring isn't the problem but that the distance between team members on an organizational chart, however, is much more important. It's ok if an external party comes in as long as the resources are treated somewhat like equals to existing engineers. But when they become "that team" you will lose collaboration and things will fall apart partly because that team's manager may have different priorities than the local team.

When it comes specifically to IT, however, we're looking at almost a race to the bottom to save money immediately, sometimes to make more budget for software engineers that are revenue centers. Furthermore, a lot of IT folks want to stay current on technologies just like many developers do, but IT constraints are oftentimes even more than what their developer peers in the same company would have. This has led to a concentration of lower performers staying for a long time (not indicating skill necessarily - their environment is hostile to professional growth) and more ambitious technical folks move elsewhere.

It's been puzzling for me why IT orgs don't focus upon automation obsessively like most industries trying hard to cut costs have. The US coal industry automated a lot of labor and it's not like coal miners were making handsome salaries like most sysadmins did in the late 90s. The low performers all seem to have a strange obsession with creating as much manual processes as possible instead of, I dunno, writing some orchestration in Rundeck jobs or Ansible playbooks maybe. One decent automation engineer can replace dozens of lower skilled junior and mid level sysadmins even in bare metal environments.

Post reply on HN