One lesson I've learned from this fiasco is to examine my own self when it comes to these situations. I am so befuddled by all the wild opinions, speculations and conclusions as well as observations of the PIR here. You can never have enough humility.
Preliminary Post Incident Review
171–180 of 227 posts
Re: Preliminary Post Incident Review
#172Earlier quoted context omitted.
If I had to guess blindly based on their writeup, it would seem that if their Content Configuration System is given invalid data, instead of aborting the template, it generates a null template. To a degree it makes sense because it's not unusual for a template generator to provide a null response if given invalid inputs however the Content Validator then took that null and published it instead of handling the null ca…
Returning null instead of throwing an exception when an error occurs is the quality of programming I see from junior outsourced developers. “if (corrupt digital signature) return null;” is the type of code I see buried in authentication systems, gleefully converting what should be a sudden stop into a shambling zombie of invalid state and null reference exceptions fifty pages of code later in some controller that’s a…
>Returning null instead of throwing an exception when an error occurs is the quality of programming I see from junior outsourced developers.
This is kernel code, most likely written in C (and regardless of language, you don't really do exceptions in the kernel at all for various reasons).
Returning NULL or ERR_PTR (in the case of linux) is absolutely one of the most standard, common, and enforced ways of indicating an error state in kernel code, across many OS's.
So it's no surprise to see the pattern here, as you would expect.
Re: Preliminary Post Incident Review
#173Earlier quoted context omitted.
> Copying my content from the duplicate thread[1] here Please don't do this! It makes merging threads a pain because then we have to find the duplicate subthreads (i.e. your two comments) and merge the replies as well. Instead, if you or anyone will let us know at hn@ycombinator.com which threads need merging, we can do that. The solution is deduplication, not further duplication!
Oops. Noted! Apologies for inadvertently adding work! Somehow, I never realized that duplicate threads were merged (instead of one of them being nuked), because it seems like a lot of work in the first place. Thanks for doing it!
Appreciated!
Re: Preliminary Post Incident Review
#174Earlier quoted context omitted.
> it sounds like they might have separate "validation" code That's what stood out to me. From the CS post: "Template Instances are created and configured through the use of the Content Configuration System, which includes the Content Validator that performs validation checks on the content before it is published." Lesson learned, a "Validator" that is not actually the same program that will be parsing/reading the fil…
I'd argue that it is completely useless. They have the actual parser that runs in production and then a separate "test parser" that doesn't actually reflect reality? Why?
Maybe they have the same parser in the validator and the real driver, but the vagaries of the C language mean that when undefined behavior is encountered, it may crash or it may work just by chance.
Re: Preliminary Post Incident Review
#175So this event is probably close to a worst case scenario for an untested sensor update. But have they never had issues with such untested updates before, like an update resulting in false positives on legitimate software? Because if they did, that should have been a clue that these types if updates should be tested too.
Re: Preliminary Post Incident Review
#176Earlier quoted context omitted.
Some industries are forced by regulation or liability to have something like crowdstrike deployed on their systems. And crowdstrike doesn't have a lot of alternatives that tick as many checkboxes and are as widely recognized.
Please give me an example of that specific regulation.
Besides things like FedRAMP mentioned in other comments, some large enterprise customers, especially banks, require terms in the contract stating the vendor uses some form of anti-malware software.
Re: Preliminary Post Incident Review
#177They bypassed the tests and staged deployment, because their previous update looked good. Ha. What if they implemented a release process, and follow it? Like everyone else does. Hackers at the workplace, sigh.
They know better obviously, transcending process and bureaucracy.
Same thing happened with Falcon on Debian before. Later they admitted that they didn't test some platforms they were releasing. Never heard of Docker?
How can you keep on with such a Q&R manager? He'll cost them billions
Re: Preliminary Post Incident Review
#178There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…
Now let's see if Microsoft listen and fixes Windows updates
Re: Preliminary Post Incident Review
#179There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…
Does anyone test their antivirus updates individually as a customer? I thought they happen multiple times a day, who has time for that?
Some sort of comprehensive test is unlikely.
But canary / smoke tests, you can do, if the vendor provides the right tools.
It's a cycle: pick the latest release, do some small cluster testing, including rollback testing, then roll out to 1%, if those machines are (mostly) still available in 5 minutes, roll out to 2%, if the 3% is (mostly) still available in 5 minutes, roll out to 4%, etc. If updates are fast and everything works, it goes quick. If there's a big problem, you'll have still have a lot of working nodes. If there's a small problem, you have a small problem.
It's gotta be automated though, but with an easy way for a person to pause if something is going wrong that the automation doesn't catch. If the pace is several updates a day, that's too much for people, IMHO.
Re: Preliminary Post Incident Review
#180I work on a piece of software that is installed on a very large number of servers we do not own. The crowd strike incident is exactly our nightmare scenario. We are extremely cautious about updates, we roll it out very slowly with tons of metrics and automatic rollbacks. I’ve told my manager to bookmark articles about the crowdstrike incident and share it with anyone who complains about how slow the update process is…
I don’t have much sympathy for CrowdStrike but deploying slowly seems mutually exclusive to protecting against emerging threats. They have to strike a balance.