Preliminary Post Incident Review
crowdstrike.com
Preliminary Post Incident Review
1–10 of 227 posts
Re: Preliminary Post Incident Review
#2"problematic content"? It was a file of all zero bytes. How exactly was that produced?
Re: Preliminary Post Incident Review
#3This reads like a bunch of baloney to obscure the real problem.
The only relevant part you need to see:
>Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data.
Problematic content? Yeah, this is telling exactly nothing.
Their mitigation is "ummm we'll test more and maybe not roll the updates to everyone at once", without any direct explanation on how that would prevent this from happening again.
Conspicuously absent:
— fixing whatever produced "problematic content"
— fixing whatever made it possible for "problematic content" to cause "ungraceful" crashes
— rewriting code so that the Validator and Interpreter would use the same code path to catch such issues in test
— allowing the sysadmins to roll back updates before the OS boots
— diversifying the test environment to include actual client machine configurations running actual releases as they would be received by clients
This is a nothing sandwich, not an incident review.
Re: Preliminary Post Incident Review
#4[deleted]
Re: Preliminary Post Incident Review
#5Copying my content from the duplicate thread[1] here:
This reads like a bunch of baloney to obscure the real problem. The only relevant part you need to see:
>Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data.
Problematic content? Yeah, this is telling exactly nothing.
Their mitigation is "ummm we'll test more and maybe not roll the updates to everyone at once", without any direct explanation on how that would prevent this from happening again.
Conspicuously absent:
— fixing whatever produced "problematic content"
— fixing whatever made it possible for "problematic content" to cause "ungraceful" crashes
— rewriting code so that the Validator and Interpreter would use the same code path to catch such issues in test
— allowing the sysadmins to roll back updates before the OS boots
— diversifying the test environment to include actual client machine configurations running actual releases as they would be received by clients
This is a nothing sandwich, not an incident review.
Re: Preliminary Post Incident Review
#6They bypassed the tests and staged deployment, because their previous update looked good. Ha.
What if they implemented a release process, and follow it? Like everyone else does. Hackers at the workplace, sigh.
Re: Preliminary Post Incident Review
#7This reads like a bunch of baloney to obscure the real problem. The only relevant part you need to see: >Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data . Problematic content ? Yeah, this is telling exactly nothing. Their mitigation is "ummm we'll test more and maybe not roll the updates to everyone at once", without any direct exp…
>Add additional validation checks to the Content Validator for Rapid Response Content. A new check is in process to guard against this type of problematic content from being deployed in the future.
>Enhance existing error handling in the Content Interpreter.
They did write that they intended to fix the bugs in both the validator and the interpreter. Though it's a big mystery to me and most of the comments on the topic how an interpreter that crashes on a null template would ever get into production.
Re: Preliminary Post Incident Review
#8"problematic content"? It was a file of all zero bytes. How exactly was that produced?
If I had to guess blindly based on their writeup, it would seem that if their Content Configuration System is given invalid data, instead of aborting the template, it generates a null template.
To a degree it makes sense because it's not unusual for a template generator to provide a null response if given invalid inputs however the Content Validator then took that null and published it instead of handling the null case as it should have.
Re: Preliminary Post Incident Review
#9This reads like a bunch of baloney to obscure the real problem. The only relevant part you need to see: >Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data . Problematic content ? Yeah, this is telling exactly nothing. Their mitigation is "ummm we'll test more and maybe not roll the updates to everyone at once", without any direct exp…
> fixing whatever made it possible for "problematic content" to cause "ungraceful" crashes
Better not only fix this specific bug but continuously use fuzzing to find more places where external data (including updates) can trigger a crash (or worse RCE)
Re: Preliminary Post Incident Review
#10This reads like a bunch of baloney to obscure the real problem. The only relevant part you need to see: >Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data . Problematic content ? Yeah, this is telling exactly nothing. Their mitigation is "ummm we'll test more and maybe not roll the updates to everyone at once", without any direct exp…
>Add additional validation checks to the Content Validator for Rapid Response Content. A new check is in process to guard against this type of problematic content from being deployed in the future. >Enhance existing error handling in the Content Interpreter. They did write that they intended to fix the bugs in both the validator and the interpreter. Though it's a big mystery to me and most of the comments on the topi…
>They did write that they intended to fix the bugs
I strongly disagree.
Add additional validation and enhance error handling say as much as "add band-aids and improve health" in response to a broken arm.
Which is not something you'd want to hear from a kindergarten that sends your kid back to you with shattered bones.
Note that the things I said were missing are indeed missing in the "mitigation".
In particular, additional checks and "enhanced" error handling don't address:
— the fact that it's possible for content to be "problematic" for interpreter, but not the validator;
— the possibility for "problematic" content to crash the entire system still remaining;
— nothing being said about what made the content "problematic" (spoiler: a bunch of zeros, but they didn't say it), how that content was produced in the first place, and the possibility of it happening in the future still remaining;
— the fact that their clients aren't in control of their own systems, have no way to roll back a bad update, and can have their entire fleet disabled or compromised by CrowdStrike in an instant;
— the business practices and incentives that didn't result in all their "mitigation" steps (as well as steps addressing the above) being already implemented still driving CrowdStrike's relationship with its employees and clients.
The latter is particularly important. This is less a software issue, and more an organizational failure.
Elsewhere on HN and reddit, people were writing that ridiculous SLA's, such as "4 hour response to a vulnerability", make it practically impossible to release well-tested code, and that reliance on a rootkit for security is little more than CYA — which means that the writing was on the wall, and this will happen again.
You can't fix bad business practices with bug fixes and improved testing. And you can't fix what you don't look into.
Hence my qualification of this "review" as a red herring.