Live data from Hacker News

UK air traffic control meltdown

jameshaydon.github.io

121–130 of 459 posts

Re: UK air traffic control meltdown

#121

Did the creator of the flight plan software engage in adversarial testing to see if they could break the system with badly formed flight plans? Or was / is the typical practice to mostly just see if the system meets just the "well-behaved" flight plan processing requirements? (with unit tests, etc)

I think we all know the answer to this.

A huge portion of "exploits" in the last 20 years have been "internal business APIs" if you will being exposed to malicious actors.

Re: UK air traffic control meltdown

#122

Earlier quoted context omitted.

Correct. The other "leg" of a solution to this problem would be to codify migration practices so stagnation at the tech level is a non issue long-term.

> codify migration practices I think this won't work: no one really wants to touch a system that works, and people will try to find any excuse to avoid migrating. The reason of this is that everyone prefers systems that work and fails in known way rather new systems that no one knows how can it fail.

Does the system work if it randomly fails and collapses the entire system for days?

People generally prefer to be lazy and to not use their brains, show up, and receive a paycheck for the minimum amount of effort. Not to be rude, but that's where this attitude originates. Having a codified process means that attitude can't exist because you're given all of the tools you need to solve the problem.

Re: UK air traffic control meltdown

#123
post #32

Earlier quoted context omitted.

Because they hit "unknown error" and when that happens on safety critical systems you have to assume that all your system's invariants are compromised and you're in undefined behavior -- so all you can do is stop. Saying this should have been handled as a known error is totally reasonable but that's broadly the same as saying they should have just written bug free code. Even if they had parsed it into some structure…

That's like saying that because one browser tab tried to parse some invalid JSON then my whole browser should crash.

You don't know that the JSON is invalid. Maybe the JSON is perfect and your parser is broken.

Re: UK air traffic control meltdown

#124

Earlier quoted context omitted.

> codify migration practices I think this won't work: no one really wants to touch a system that works, and people will try to find any excuse to avoid migrating. The reason of this is that everyone prefers systems that work and fails in known way rather new systems that no one knows how can it fail.

Does the system work if it randomly fails and collapses the entire system for days? People generally prefer to be lazy and to not use their brains, show up, and receive a paycheck for the minimum amount of effort. Not to be rude, but that's where this attitude originates. Having a codified process means that attitude can't exist because you're given all of the tools you need to solve the problem.

> Having a codified process means that attitude can't exist because you're given all of the tools you need to solve the problem.

Yes, but in real life doesn't work. Processes have corner cases. As you said, people are lazy and will do everything to find the corner case to fit in.

Just an example from the banking sector. There are processes (and even laws) that force banks to use only certified, supported and regularly patched software: there are still a lot of Windows 2000 servers in their datacenters and will be there for many years.

Re: UK air traffic control meltdown

#125
post #86
post #58

Earlier quoted context omitted.

> Even if they had parsed it into some structure this would be the equivalent of a KeyError popping out of nowhere because the code assumed an optional key existed. How many KeyError exceptions have brought down your whole server? It doesn't happen because whoever coded your web framework knows better and added a big try-catch around the code which handles individual requests. That way you get a 500 error on the spec…

Crash is a feature, though. It's not like exceptions raises by itself into interpreter specifications. It's just that it so happens that Web apps ain't need no airbags that slow down businesses.

That line of reasoning is how you have systemic failures like this (or the Ariane 5 debacle). It only makes sense in the most dire of situations, like shutting down a reactor, not input validation. At most this failure should have grounded just the one affected flight rather than the entire transportation network.

Re: UK air traffic control meltdown

#126
Trusted input rarely should be trusted. It's input. You need to validate it as if it is hostile and have a process for dealing with malformed input. Now of course, standing by the sidelines it is easy to criticize and I'm sure whoever worked on this wasn't stupid. But I've seen this error often enough now in practice that I think that it needs to be drilled into programmers heads more forcefully: stuff is only valid if you have just validated it. If you send it to someone else, if someone you trust sends it to you, if you store in a database and then retrieve it and so on then it is just input all over again and you probably should validate it for being well-formed. If you don't do that then you're a bitflip, migration or an update away from an error that will cause your system to go into an unstable state and the real problem is that you might just propagate the error downstream because you didn't identify it.

Input is hard. Judging what constitutes 'input' in the first place can be harder.

Re: UK air traffic control meltdown

#127

Earlier quoted context omitted.

That's like saying that because one browser tab tried to parse some invalid JSON then my whole browser should crash.

No, it's more like saying your browser has detected possible internal corruption with, say, its history or cookies database and should stop writing to it immediately. Which probably means it has to stop working.

It definitely isn't. It was just a validation error in one of thousands external data files that the system processes. Something very routine for almost any software dealing with data.

Re: UK air traffic control meltdown

#128

Earlier quoted context omitted.

Presumably you need to know where upcoming flights are going to be in the future (based on the plan), before they hit radar etc.

For the most part (although there are important exceptions), IFR flights are always in radar contact with a controller. The flight plan is tool allows ATC and the plane to agree a route so that they don't have to be constantly communicating. ATC 'clears' a plane to continue on the route to a given limit, and expects the plane to continue on the plan until that limit unless they give any future instructions. In this r…

Mostly yes; however, there are large parts of the Atlantic and Pacific where that isn't true (radar contact). I know the Atlantic routes are frequently full of plans that left the US and Canada heading to the UK.

I have no idea what percent of the volume into the UK comes from outside radar control; if they asked a flight to divert, that may open multiple other cans of worms.

Re: UK air traffic control meltdown

#129
post #74

Earlier quoted context omitted.

It won't fix anything. JSON is the "standard" today, 15 years ago it was XML and in 15 years we will have protobuf or another new standard.

Correct. The other "leg" of a solution to this problem would be to codify migration practices so stagnation at the tech level is a non issue long-term.

You could do all that stuff.

But after you did it, you'd still have exactly the same problem. The cause was not related to deserialization. That part worked perfectly. The problem is the business logic that applied to the model after the message was parsed.

Re: UK air traffic control meltdown

#130
post #49
post #43

Earlier quoted context omitted.

Maybe he doesn’t care if people stop reading and he’d prefer to use the language he’s most comfortable with? It’s his blog after all, not yours. Additionally, perhaps he’s making the point that a language with an expressive type system makes solving problems like this trivial.

If you don't care about readers reading it or not then what is the point to publish an article ?

Why does there have to be a point? If there is one, why do you need to understand it?
Post reply on HN