Live data from Hacker News

British Airways: All flights cancelled amid IT crash

bbc.com

21–30 of 255 posts

Re: British Airways: All flights cancelled amid IT crash

#21
I have zero evidence, and am simply working through a though exercise, but it could be targeted?

Maybe a group on behalf of a nation state is testing out its muscle. Or sending a warning. Maybe the UK did something recently a nation didn't like, and this is the new form of "diplomatic protest". Or sending a warning shot.

I remember reading last year that the large Delta airlines outage was cyber terror related.

It really feels like we're not too far off from a war between two nations without a single bullet fired. What happens when a country is hit with nation level ransomware? Ie not, "give us $300 bucks and you'll get your PC back", but "sign XYZ treaty and you get your country's water system back"? Or "we'll restore your internet and turn your powerplants back online"? How much will a wealthy country (like us in the US) be willing to stomach of seeing people going thirsty and hungry before they want their govt to capitulate?

Of course the world will condemn and complain, but as has always been the case the country with the largest Army makes the rules. And we might be using the wrong measuring stick. Microsoft missed the boat by thinking (along with most of the industry) that measuring computing meant measuring PCs. It wasn't until it was too late that it realized that computing was about to be dominated by Mobile and smaller. They were using the wrong measuring stick.

Are we on the verge of an era where an army should be measured in its digital strength and not its physical strength? It's a very scary thought.

As an American I think first of the risks to my own country. How long could we sit with a nationwide blackout, and the internet down (for anyone using backup generators)? Food rotting in shipping containers because no cranes can offload it. No easy communication or flights for quick way to move people or things around.

As a country (and as a world) I can't help but feel we're really not doing enough to take a risk like this seriously. It's just a feeling, but based on small incidents here and there it really feels to me like something in the next 5-10 years is gonna happen that'll make us all drop our jaws and say "I didn't think this was possible."

Regardless of your political beliefs, a lot of people around the world didn't think last year's US election was possible. I was shocked by it as well, but it also opened my eyes to how much more really is in the realm of possible.

As engineers we have much a deeper knowledge of the risks involved and as a result a much greater responsibility of raising awareness and getting the problem fixed.

EDIT: I'm not sure of the original location I saw the Delta stuff, but a quick look now turns up this link.

http://observer.com/2016/09/did-a-cyber-attack-ground-delta-...

And yes, I'll repeat this above comment is purely speculative by me.

Re: British Airways: All flights cancelled amid IT crash

#22

I have zero evidence, and am simply working through a though exercise, but it could be targeted? Maybe a group on behalf of a nation state is testing out its muscle. Or sending a warning. Maybe the UK did something recently a nation didn't like, and this is the new form of "diplomatic protest". Or sending a warning shot. I remember reading last year that the large Delta airlines outage was cyber terror related. It re…

There's a little insider knowledge on the Flyertalk forum.

See this post, for example...a BA employee: https://www.flyertalk.com/forum/28366141-post168.html

Later, there's some talk about a power outage and/or lighting strike, again from BA employees.

Re: British Airways: All flights cancelled amid IT crash

#23

I have zero evidence, and am simply working through a though exercise, but it could be targeted? Maybe a group on behalf of a nation state is testing out its muscle. Or sending a warning. Maybe the UK did something recently a nation didn't like, and this is the new form of "diplomatic protest". Or sending a warning shot. I remember reading last year that the large Delta airlines outage was cyber terror related. It re…

> I remember reading last year that the large Delta airlines outage was cyber terror related.

Where'd you hear that? AFAIK, it was a botched failover to backup power.

https://www.theregister.co.uk/2016/08/08/computer_fault_take...

Re: British Airways: All flights cancelled amid IT crash

#24
post #22

I have zero evidence, and am simply working through a though exercise, but it could be targeted? Maybe a group on behalf of a nation state is testing out its muscle. Or sending a warning. Maybe the UK did something recently a nation didn't like, and this is the new form of "diplomatic protest". Or sending a warning shot. I remember reading last year that the large Delta airlines outage was cyber terror related. It re…

There's a little insider knowledge on the Flyertalk forum. See this post, for example...a BA employee: https://www.flyertalk.com/forum/28366141-post168.html Later, there's some talk about a power outage and/or lighting strike, again from BA employees.

I could be mistaken, but from what I'm seeing in that thread they really have no idea what caused it. Which, btw, seems fairly reasonable for how early into this incident it is. I wouldn't expect a root cause for a while.

Delta Airlines also called their outage a "power outage", but it looks like there was reason to believe it wasn't.

http://observer.com/2016/09/did-a-cyber-attack-ground-delta-...

Re: British Airways: All flights cancelled amid IT crash

#25

One customer was told that the root cause was a "lightning strike on a datacentre", although it sounds pretty unlikely (there are storms today in the UK but surely there'd be a DR plan?)

They do have DR plans, but the reality is that whole thing is a huge distributed system with parts from different vendors, in different locations, etc, with decades of legacy.

A power outage the reboots a whole datacenter would screw any major airline for at least half a day.

Thus far, nobody has found the economics of a modern, truly HA setup worth the cost. Outsiders greatly underestimate the complexity too. Think something like 40 disparate applications from different vendors, or some homegrown systems, in different geographical locations. Then, all the client applications are in buildings you don't own (airports) where you aren't allowed to control the infrastructure.

If it were a high margin business, perhaps things would be different. It's not.

Re: British Airways: All flights cancelled amid IT crash

#27

The main KPI for BA's IT function is the percentage of jobs they have moved "nearshore" (to Krakow). Not that I think this is the Krakow workers' fault, mind you; rather this (and the string of similar IT incidents recently at BA) seems to fit the pattern of upper management focusing on driving down costs to the exclusion of everything else.

Krakow is a big tech center, is in the EU, and it's not obvious why moving jobs there would have anything to do with a drop in quality.

Re: British Airways: All flights cancelled amid IT crash

#28
post #22

Earlier quoted context omitted.

There's a little insider knowledge on the Flyertalk forum. See this post, for example...a BA employee: https://www.flyertalk.com/forum/28366141-post168.html Later, there's some talk about a power outage and/or lighting strike, again from BA employees.

I could be mistaken, but from what I'm seeing in that thread they really have no idea what caused it. Which, btw, seems fairly reasonable for how early into this incident it is. I wouldn't expect a root cause for a while. Delta Airlines also called their outage a "power outage", but it looks like there was reason to believe it wasn't. http://observer.com/2016/09/did-a-cyber-attack-ground-delta-...

I don't put much stock in that article. Their conspiracy theory is that no other businesses saw a power outage. But it didn't have to be a supplier/mains problem. They could have had an issue with their own, internal power grid. Their data center likely has a UPS, generators, huge switches to move back and forth from battery/mains/generator etc. A fault there would cause an outage.

Edit: Yep. http://bgr.com/2016/08/14/delta-finally-explained-how-one-po...

Re: British Airways: All flights cancelled amid IT crash

#29
post #22

I have zero evidence, and am simply working through a though exercise, but it could be targeted? Maybe a group on behalf of a nation state is testing out its muscle. Or sending a warning. Maybe the UK did something recently a nation didn't like, and this is the new form of "diplomatic protest". Or sending a warning shot. I remember reading last year that the large Delta airlines outage was cyber terror related. It re…

There's a little insider knowledge on the Flyertalk forum. See this post, for example...a BA employee: https://www.flyertalk.com/forum/28366141-post168.html Later, there's some talk about a power outage and/or lighting strike, again from BA employees.

As a data point, there was an intense lightning storm in south west UK (where I live atm) around 3am this morning (~13 hours ago).

There were lightning flashes a few times per second, near continuously... for about 1 1/2 hours. I've never seen anything like it before.

Re: British Airways: All flights cancelled amid IT crash

#30

One customer was told that the root cause was a "lightning strike on a datacentre", although it sounds pretty unlikely (there are storms today in the UK but surely there'd be a DR plan?)

that would make a lot of sense - there was a heavy storm that moved north from south coast right at that time

you can have all the DR in the world but if you never fully test it - you never know if it's going to be 100% on the day - the only way to fully test is to do a full failover from the primary and see what happens - but it could be a real career low if you do that on something so commercially sensitive and it all goes very wrong

interestingly it's a bank holiday weekend - so every possibility some people who knew some of the special incantations to get things back up on days like this are out camping somewhere remote - and will have 500+ messages when they get back in to signal tomorrow and wonder why they ever try to go away - and if they ever do go away again - wonder how they will ever be able to relax

Post reply on HN