Earlier quoted context omitted.
This is by design, it meets the primary goal of the government which is funding patronage jobs that make few, if any, demands on the worker.
[flagged]
[deleted]
A corrupt file led to the FAA ground stoppage – also found in backup system
171–180 of 402 posts
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#172In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay". So, if you have a database of some sort, a part of the backup process would be checking that the backup can be used to run an instance of it. If you have images or videos or PDF files, all of those should be validated as well, to make sure that they're not c…
> In the past years, I've grown increasingly concerned about backups. To me, it feels like whatever you back up should be validated, before it's considered "okay".
I've been trying to toot this horn at every place I've consulted/worked at, with the most common response from software engineers being: "We're using the AWS backup system of the database, of course it works, no need to test it"
People have way too much trust in systems overall, especially ones they have no inside information about. And people also seems to default to "Of course it works" without testing it, instead of "How would I know it works if I haven't tested it?" or even "I don't know".
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#173Earlier quoted context omitted.
This is common for government agencies. I worked at Labor and your second paragraph hit very close to home. The only plus side was the insane amount of free time you spent waiting. I tried to go back on the civilian side of the contracts and I nope'd out of it as soon as I hit red tape
This is by design, it meets the primary goal of the government which is funding patronage jobs that make few, if any, demands on the worker.
All the federal employees I have met are EXCELLENT. It is very difficult to get hired into one of the dwindling jobs at the agencies. A lot of the government has been contracted out. Having worked at a federal IT contractor, I can say that in my experience most of those workers are very good and dedicated to their work. HOWEVER, they don’t always understand well the mission at hand. Some of this is to be expected given that they are contractors who come and go more frequently than federal employees charged with implementing government programs.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#174It seems like every year or so we have some gigantic technology meltdown in this industry. Imagine something like this happening at the NYSE, CME, et. at. Or, simply think about the last time you heard about a nationwide credit/debit card outage... Why can't we have our national infrastructure systems running at least as reliably as the Amex network? These systems are all information clearinghouses at the end of the…
All of those systems have outages all the time, mostly isolated incidents when it comes to NYSE, CME and similar (here is the outage page for NYSE for example: https://www.nyse.com/market-status/history). Visa, Mastercard and others have nationwide outages from time to time too, they're in no way invincible like you seem to imply. Latest Visa outage I can remember must have been around 2018/2019 sometime, but probably it happened later than that too, except I didn't notice it then.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#175It seems like every year or so we have some gigantic technology meltdown in this industry. Imagine something like this happening at the NYSE, CME, et. at. Or, simply think about the last time you heard about a nationwide credit/debit card outage... Why can't we have our national infrastructure systems running at least as reliably as the Amex network? These systems are all information clearinghouses at the end of the…
How many people do you thing will get fired due to the grounding of airplanes?
versus: how many people would get fired if NYSE went down due to "we didn't think we needed backups".
incentives matter. In political systems the main goal is to be able to point the finger at somebody else, not to ensure things run well.
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#176Earlier quoted context omitted.
This is by design, it meets the primary goal of the government which is funding patronage jobs that make few, if any, demands on the worker.
[flagged]
Here here. Also, if there weren’t red tape, then there would be more corruption and lack of accountability. The staff of agencies are damned if they do, damned if they don’t. If one wants to critique government agencies, criticize the political appointees who are in thrall to the industries they are supposed to be regulating. The rank and file generally work hard and in good faith. They are just trying to be good stewards of public resources. I’ve seen this at the federal level and state levels (primarily in North Carolina and Louisiana).
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#177Earlier today another HN user linked to a PDF from a previous 2018 (cira 2014) investigation that pointed to the "dual-channel back up" system being fragile and likely insufficient. https://news.ycombinator.com/item?id=34338373 > ERAM’s original design did not include a dedicated backup system. FAA believed that ERAM did not need one due to the redundancy provided by the system’s dual channel design. This design was…
Turns out having 2 copies of something doesn't matter if same message re-tried after sending to first node crashes another node, who could possibly predict that /s
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#178Earlier quoted context omitted.
I see this a lot with people are experts in real time operating systems environments, particularly in aviation/space stuff (maybe because that’s where I worked for a while). They have excellent intuition around making things redundant to single pieces of hardware failing but don’t really grok making stuff resilient to wider failures. Anything involving transaction logs, rollbacks, and plain old backups take a backsea…
Formal verification of not having such fatal bugs would allow a real-time system without reliance of backups to not screw up, but still of course having logs/rollbacks for (human) input actions to cope with mistakes. It's just that production software essentially never used formal verification to a sufficient extend.
Well, it is hideously expensive to do on bigger pieces of code. And I'd imagine you can still get the spec you verify against wrong
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#179Earlier quoted context omitted.
This seems to pose an interesting question that's out of my pay grade. The fundamental problem seems to be: you've replaced two distinct systems (one new and far more capable + 1980s-era one that always works but lacks [new feature x100]) with the same one running on 2x different machines. So the weak point is you ultimately share the same database/data structures/memory+logic flows between two systems. So if you kee…
My intuition on that is that two is also a bad number to choose in that case. One could fo full lunar mission on the thing and have three models and in case of inconsistency the majority wins.
It is but it doesn't matter if it is the input message that crashes it. You just need 3 bad message (sooo a message and 2 retries) to crash it whole
Re: A corrupt file led to the FAA ground stoppage – also found in backup system
#180Earlier quoted context omitted.
This seems to pose an interesting question that's out of my pay grade. The fundamental problem seems to be: you've replaced two distinct systems (one new and far more capable + 1980s-era one that always works but lacks [new feature x100]) with the same one running on 2x different machines. So the weak point is you ultimately share the same database/data structures/memory+logic flows between two systems. So if you kee…
My intuition on that is that two is also a bad number to choose in that case. One could fo full lunar mission on the thing and have three models and in case of inconsistency the majority wins.
Ideally three models developed to the same spec independently, with majority voting