Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

391–400 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#391
post #254

Earlier quoted context omitted.

The problem is that spelling out complicated things is hard. Take the law for example - in theory we have a coherent code that specifies exactly what things are crimes and the appropriate methods of dealing with them. In practice, it takes teams of highly trained professionals and an elaborate system of courts to clarify what these laws mean in all but the most trivial cases. Generally you need some flexibility to ha…

Which legal theory postulates a ‘coherent code that specifies exactly what things are crimes and the appropriate methods of dealing with them’?

That is a different use of the word theory, and serves as an excellent example of why the problem of communicating complex ideas so unambiguously as to eliminate the need for interpretation is so intractable.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#392

Earlier quoted context omitted.

You act like "moving fast" is all you need to know to "move fast". As if it's simply the skill of making time move faster, and you don't need any other skills than that, because once you've broken the laws of physics and changed the speed of time, everything just works faster without any differences or consequences. Do you watch a lot of Superhero movies? You should work smarter, not harder. Just turn up your smart k…

I'm not sure what point you are trying to make. Physics? super heroes? knobs?

My point is that "you never learned how to move fast" is just as glib and patronizing and useless advice as "work smarter, not harder".

ESPECIALLY if millions of people's lives and fortunes are at stake, as in this case with air traffic control.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#393
post #358

Earlier quoted context omitted.

When you are in a complex spiderweb you simply cannot move fast. If you are moving fast you are not looking at everything and it will blow up in your face.

True.. which is why you want simple systems that are easy to understand and iterate on.

The worldwide air traffic control system is not simple or easy to understand and iterate on. And that's not because it was designed by idiots, or that you're so much smarter and more experienced and a vastly better programmer than the combined efforts of everyone in the world working on air traffic control, as you seem to be implying from your comfortable armchair.

Are you proposing the entire world simply give up air travel, because government regulations and industry standards and the laws of physics and chaos theory prevent you from having the simple easy to understand air traffic control system you envision?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#394

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

The only way to optimize for lowest overall risk is to optimize for speed of change. All the checklists in the world to prevent something from happening are fine and dandy until something happens anyway (which it will). And then they hamstring you from actually fixing it. Instead, if you can move fast consistently, you can minimize the total downtime.

Please don't change jobs from Facebook to the air traffic control industry.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#395
post #287

Earlier quoted context omitted.

Two examples: 1. The FAA basically handed their risk-management keys over to Boeing when authorizing the 737-MAX, contributing to those deaths ( https://www.newyorker.com/news/our-columnists/how-boeing-and... ) 2. The FAA's pilot medical vetting process, while thorough, is behind the times. There are people who took ADHD medicine in high school that are unable to obtain a medical certificate due to the FAA's overly-s…

The FAA was gutted in the name of deregulation and competition - you might want to ask the GOP what happens when you don't have adequate govt oversight and regulation. Two planes crashed due to a design flaw - hundreds of people killed and one manufacturer and model forever tarnished like McDonald Douglass and their DC-10.

And you might also want to ask the GOP why Ronald Reagan fired 11,345 air traffic controllers and banned them from federal service for life.

https://en.wikipedia.org/wiki/Professional_Air_Traffic_Contr...

>On August 5, following the PATCO workers' refusal to return to work, the Reagan administration fired the 11,345 striking air traffic controllers who had ignored the order, and banned them from federal service for life. In the wake of the strike and mass firings, the FAA was faced with the difficult task of hiring and training enough controllers to replace those that had been fired. Under normal conditions, it took three years to train new controllers. Until replacements could be trained, the vacant positions were temporarily filled with a mix of non-participating controllers, supervisors, staff personnel, some non-rated personnel, military controllers, and controllers transferred temporarily from other facilities. PATCO was decertified by the Federal Labor Relations Authority on October 22, 1981. The decision was appealed but to no avail, and attempts to use the courts to reverse the firings proved fruitless.

My late friend Ron Reisman worked at NASA Ames Research Center on air traffic control and flight safety, and he hired up a bunch of the professional air traffic controllers who Reagan fired, and taught them to program.

Because it's much easier to teach an air traffic controller how to program, than it is to teach a programmer how to control air traffic.

And we have them to thank for how safe the air traffic control system is today.

https://www.nasa.gov/50th/Folklife/biosPropulsion.html

>Ron Reisman has BA in Philosophy and Classical Greek, and an MS in Computer Science. He joined NASA Ames Research Center in 1988 as one of the original members of the Center Tracon Automation System development team. Since the late 1990s he has worked on traffic flow management research and development. He is currently supporting the Next Generation Air Traffic System research.

I saw him give an earlier version of this talk at the November 1989 Usenix Montery Graphics Conference, where he discussed training air traffic controllers to program, and he subsequently gave me a tour of the flight simulators and air traffic control systems at NASA Ames:

https://www.usenix.org/legacy/events/usenix02/usenix02.pdf

>INTRODUCTION TO AIR TRAFFIC MANAGEMENT SYSTEMS

>Ron Reisman and James Murphy, NASA Ames Research Center, and Rob Savoye, Seneca Software

>This introduction to air traffic control systems summarizes the operational characteristics of the principal Air Traffic Management (ATM) domains (i.e., en route, terminal area, surface control, and strategic traffic flow management) and the challenges of designing ATM decision support tools. The Traffic Flow Automation System (TFAS), a version of the Center TRACON Automation System (CTAS), will be examined. TFAS achieves portability across platforms (Solaris, HP/UX, and Linux) by adherence to software standards (ANSI, ISO, POSIX). Software engineering issues related to design, code reuse, portability, performance, and implementation are discussed.

Based on what I know about Ron's and other people's diligent methodological work on air traffic control and safety, I feel extremely safe and confident flying, and I find it insulting to his memory and the legacy of his work when the armchair architect ex-Facebook employees on this thread (and the GOP) glibly and patronizingly implore the FAA to "move fast and break things", as if they had no idea how many lives and fortunes are at stake.

Here's a video of Ron showing Marvin Minsky the flight simulator, an early AR headset, and the hydraulic lifts:

https://www.youtube.com/watch?v=mOKENF_-z8Y

I previously mentioned his earlier work making Apple ]['s talk with dolphins, and his thoughts on AI and dolphin intelligence:

https://news.ycombinator.com/item?id=32039126

To answer "What is AI?" you first have to answer "What is I?"

Check out the Apple ][ at the Dolphin Research Center in 1982, and Ron Reisman’s thoughts about dolphin intelligence:

https://www.youtube.com/watch?v=CWHCTNztnwQ&t=392s

“It’s gotten to the point that I never say anything about intelligence in general. I don’t know what it means any more. I used to. But then I started trying to test it. And if you think about it for a while, you don’t know what it is.” -Ron Reisman

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#396

Earlier quoted context omitted.

I see this a lot with people are experts in real time operating systems environments, particularly in aviation/space stuff (maybe because that’s where I worked for a while). They have excellent intuition around making things redundant to single pieces of hardware failing but don’t really grok making stuff resilient to wider failures. Anything involving transaction logs, rollbacks, and plain old backups take a backsea…

> They have excellent intuition around making things redundant to single pieces of hardware failing but don’t really grok making stuff resilient to wider failures. I always feel like making single components redundant is a fairly well-defined process -- generally speaking, the mechanisms are the same (1+ redundant components, failover, STONITH, etc), where making things resilient on a higher level is not as well-defi…

Hmm?

BFT state machine replication is well-understood and well-defined: use N of M agreement for inputs and run them through a deterministic state machine. Optionally, do N of M signature of outputs.

OTOH what are properties of failover? "Failover" seems like an attempt to cheat on Byzantine generals' problem: Generals send mail and the confirm results in a Zoom call. But what if Zoom doesn't work? What are the assumptions for 1+ redundant components/failover/STONITH?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#397

Earlier quoted context omitted.

The FAA was gutted in the name of deregulation and competition - you might want to ask the GOP what happens when you don't have adequate govt oversight and regulation. Two planes crashed due to a design flaw - hundreds of people killed and one manufacturer and model forever tarnished like McDonald Douglass and their DC-10.

And you might also want to ask the GOP why Ronald Reagan fired 11,345 air traffic controllers and banned them from federal service for life. https://en.wikipedia.org/wiki/Professional_Air_Traffic_Contr... >On August 5, following the PATCO workers' refusal to return to work, the Reagan administration fired the 11,345 striking air traffic controllers who had ignored the order, and banned them from federal service for l…

737Max saga has nothing to do with air traffic control.

"The FAA, citing lack of funding and resources, has over the years delegated increasing authority to Boeing to take on more of the work of certifying the safety of its own airplanes."

"There wasn’t a complete and proper review of the documents,” the former engineer added. “Review was rushed to reach certain certification dates.”

Seems like FAA certification was a disaster in the making.

https://www.seattletimes.com/business/boeing-aerospace/faile...

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#398
post #200
post #73

Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…

The NOTAM system is something that a room full of decent engineers could easily build from scratch and make it infinitely better in a short time. It’s essentially just a database of categorized posts with some APIs for sending entries and and returning them when requested. These government IT teams spend way more than what it should cost and end up with bloated ancient tech that barely works. That’s not speaking poor…

Engineers following current best practices? They'd run a Kubernetes cluster with Kafka. Because those are the best CotS tool for reliability. Battle-tested. The system will be down every week because Kubernetes need patching.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#399
post #391

Earlier quoted context omitted.

Which legal theory postulates a ‘coherent code that specifies exactly what things are crimes and the appropriate methods of dealing with them’?

That is a different use of the word theory, and serves as an excellent example of why the problem of communicating complex ideas so unambiguously as to eliminate the need for interpretation is so intractable.

More concretely, I have never read or encountered anyone educated in or practicing law who supposed this could even exist.

Where did you acquire this notion?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#400

Earlier quoted context omitted.

True.. which is why you want simple systems that are easy to understand and iterate on.

The worldwide air traffic control system is not simple or easy to understand and iterate on. And that's not because it was designed by idiots, or that you're so much smarter and more experienced and a vastly better programmer than the combined efforts of everyone in the world working on air traffic control, as you seem to be implying from your comfortable armchair. Are you proposing the entire world simply give up ai…

Who said anything about the worldwide air traffic control system? Did you mean to reply to a different thread?
Post reply on HN