Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A corrupt file led to the FAA ground stoppage – also found in backup system

cnn.com

Post reply on HN
Print view View original

271–280 of 402 posts

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#271

Post by alexb_ »

huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

You've got it exactly right. There are a lot of people here who are completely deluded into the "move fast and break things" mindset not realizing that sometimes you really do not want to move fast, because you REALLY do not want to break things. A corrupted file throwing up panics like this is a good thing, because you don't want corrupted files to pass through like everything is fine.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#272

Post by specialp »

greggarious wrote: post #220
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

If they’re so risk averse, why are their systems failing so severely? It’s absolutely a corruption issue, in that the government prefers to pay 2-3x what they would to solve things internally to contractors who then perform a poor job and lobby to keep whatever they build in place for decades.

I am not sure if you ever dealt with a very entrenched system that has been in place longer than you have been alive, but it is not easy. It is also not easy to hire to deal with these things. Especially at government rates which are lower than the private sector.

If you were the FAA and started up an internal startup to find only the best to replace systems that have been running forever you would face a lot of problems.

1. Nobody would give you the funding until something like what happened yesterday did

2. Your developers getting paid more than the FAA directors will get lots of political attention

3. Even a great internal engineering team would most likely take ages to do something like this. This isn't move fast and break stuff with a greenfield, it is painful deconstruction and analysis of a very complicated system.

4. Many people won't want to work on this no matter how much you are paying.

It is also out of the wheel house of an agency tasked with flight safety. So expensive contractors arise. Do they do a poor job often? Yes. But these are jobs that are really hard to scope and execute. If it is really a case of overpaid contractors coming in and not doing the work, people here should start a startup and hire a SEAL team six of 1970's computer system rip and replacers and make a lot of money.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#273

Post by justjash »

r3trohack3r wrote: post #264
greggarious wrote: post #220

Earlier quoted context omitted.

If they’re so risk averse, why are their systems failing so severely? It’s absolutely a corruption issue, in that the government prefers to pay 2-3x what they would to solve things internally to contractors who then perform a poor job and lobby to keep whatever they build in place for decades.

> It’s absolutely a corruption issue, in that the government prefers to pay 2-3x Being serious, I wish I could find one of these 2-3x multiple payouts in government. Every time I've looked at anything government related (including direct contractors) the pay is garbage. Usually 15% to 50% of what the private market pays for my skill set.

Worked for a govt contractor years ago and I would imagine the payout from the govt contract was at least 2-3x. There were usually 2-3 layers of people getting paid and I don't think any of them were hurting. Of course no employee was getting a massive payout compared to private.

Govt > contractor > sub-contractor > employee was pretty common. I never knew of anyone being a direct contractor to the agency since the contracts were so large and involved a lot of employees.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#274

Post by panzagl »

r3trohack3r wrote: post #264
greggarious wrote: post #220

Earlier quoted context omitted.

If they’re so risk averse, why are their systems failing so severely? It’s absolutely a corruption issue, in that the government prefers to pay 2-3x what they would to solve things internally to contractors who then perform a poor job and lobby to keep whatever they build in place for decades.

> It’s absolutely a corruption issue, in that the government prefers to pay 2-3x Being serious, I wish I could find one of these 2-3x multiple payouts in government. Every time I've looked at anything government related (including direct contractors) the pay is garbage. Usually 15% to 50% of what the private market pays for my skill set.

The contracting company receives usually 2-3x what the actual developer makes- so if the gov't pays $300K for a developer, they get a $100K developer. Some of that overhead is justified, but a lot of it is the company's profit.

The old answer used to be for the government to hire directly, but that's been hamstrung for like 40 years by now.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#275

Post by ThinkingGuy »

bambax wrote: post #201

> NOTAMs (Notice to Air Missions) OT, but didn't NOTAM use to mean Notice to Airmen? It's still that on the ICAO website: https://www.icao.int/safety/istars/pages/notams.aspx Did it change, or is this a CNN initiative? (And how does one notify a mission?)

The FAA's website has the updated nomenclature:

https://www.faa.gov/about/initiatives/notam/what_is_a_notam

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#276

Post by btown »

JCM9 wrote: post #231
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

This demonstrates a gross misunderstanding of how the FAA actually operates in its efforts to address “risk.” It’s far more theatre via bureaucracy and paperwork than actual proven engineering efforts that demonstrably reduce risk.

Is this conjecture or based on documented shortcomings of their approach though? Much of this paperwork, especially if created in response to NTSB crash post-mortems, could very well be examples of https://fs.blog/chestertons-fence/ - is there reason to believe otherwise?

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#277

Post by skissane »

nradov wrote: post #248
avar wrote: post #209

Earlier quoted context omitted.

We know what it's like in software because we have e.g. the example of how the Space Shuttle's flight control software was developed and audited. It's weeks of meetings, tests and change processes etc. obsessing over changing a single instruction in assembly. Which also suggests to me that the problem isn't per-se the glacial progress & approval processes, but that they're using the wrong glacial processes. If this F…

The NASA Space Shuttle flight software was among the best ever developed. There was never a defect that impacted safety. But that team had kind of an "unfair" advantage in that were able to program in assembly code on bare metal with no real software stack. Whereas the rest of us are forced to build on a foundation of sand using multiple layers of low-quality third-party software in order to deliver any useful functi…

> But that team had kind of an "unfair" advantage in that were able to program in assembly code on bare metal with no real software stack

The Space Shuttle’s avionics software was not written in assembly, rather HAL/S, a high-level language invented for the project. Assembly was mainly used for the custom real-time OS kernel. They also maintained their HAL/S toolchain, which was written in XPL-a PL/I dialect which was popular for compiler development in the 1970s. The development environment ran on IBM mainframes, and the main CPUs on the Shuttles were the aerospace derivatives of the IBM S/360 mainframe architecture, System/4pi, model AP-101. The same CPUs were used by USAF (e.g. the B-1 bomber), but USAF mainly used JOVIAL to program theirs. Another big user of JOVIAL was the FAA, who used it to write a lot of their original mainframe-based air traffic control software (FAA HOST).

The Space Shuttle team inventing their own programming language was a byproduct of the time the project started (1970s). If they’d started a decade later, they probably would have used Ada instead. But Ada didn’t exist yet, and they thought inventing their own language was a better choice than JOVIAL

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#278

Post by brazzy »

gnarbarian wrote: post #257
malfist wrote: post #228

Earlier quoted context omitted.

Which is why we've not had a commercial plane crash since.......2009? Seems FAA is pretty effective at reducing risk.

a few things 1) mostly thank the airline operators for that. 2) the fact the 737 Max disasters happened outside the US was a matter of chance not exemplary policies by the FAA. FAA policies did not prevent the deployment and roll out of this dangerous aircraft in the United States and did not lead to grounding of the aircraft until multiple events occurred. 3) I'm mostly talking about software here, and I believe a b…

> 1) mostly thank the airline operators for that.

This is a 100% proof that you have not the faintest clue what you're talking about.

The #1 priotity of airline operators is profit, not safety.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#279

Post by haswell »

JCM9 wrote: post #231
huslage wrote: post #219

You all seem to think this is similar in value or operation to a web app. It is not. It is a safety-critical system that requires very stringent operational and development guidelines ON PURPOSE. The idea that the FAA shouldn't be risk averse in this system is absolutely ridiculous. The complexity of operating the airspace of an entire nation is nothing to scoff at and the importance of the NOTAM system should not be…

This demonstrates a gross misunderstanding of how the FAA actually operates in its efforts to address “risk.” It’s far more theatre via bureaucracy and paperwork than actual proven engineering efforts that demonstrably reduce risk.

This is a sweeping claim with really nothing to back it up. Are you saying this from an inside knowledge of the FAA, or is this just an opinion?

On the surface, the relative safety of air travel and the lack of major stoppages over a span of 22 years seems like a major counter example.

You’re making this statement emphatically and authoritatively, though, so I’m curious to understand where that certainty comes from and how it accounts for the other publicly visible properties of the FAA and air travel.

Re: A corrupt file led to the FAA ground stoppage – also found in backup system

#280

Post by tyingq »

defrost wrote: post #117
SSLy wrote: post #115

Earlier quoted context omitted.

Could be a Solaris.

Given the vintage pedigree BSD like SunOS is an even better candidate. Once tuned for processing they just ran - rip out a network cable and stuff it back in again, not a hiccup. Many of the time saw no benefit in porting forward to Slow Loris.

I don't think weblogic exists for SunOS.
Post reply on HN