[flagged]
Preliminary Post Incident Review
51–60 of 227 posts
Re: Preliminary Post Incident Review
#52Why do they insist on using what sounds like military pseudo jargon throughout the document? ex. sensors? I mean how about hosts, machines, clients?
It’s endemic in the tech security industry - they’ve been mentally colonised by ex-mil and ex-law enforcement (wannabe mil) folks for a long time. I try to use social work terms and principles in professional settings, which blows these people’s minds. Advocacy, capacity evaluation, community engagement, cultural competencies, duty of care, ethics, evidence-based intervention, incentives, macro-, mezzo- and micro-pra…
Even computer security itself is a metaphor (at least in its inception). I often wonder what if instead of using terms like access, key, illegal operation, firewall, etc. we'd instead chosen metaphors from a different domain, for example plumbing. I'm sure a plumbing metaphor could also be found for every computer security concern. Would be so quick to romanticize as well as militarize a field dealing with "leaks," "blockages," "illegal taps," and "water quality"?
Re: Preliminary Post Incident Review
#53Lots of words about improving testing of the Rapid Response Content, very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes". > Enhance existing error handling in the Content Interpreter. That's it. Also, it sounds like they might have separate "validation" code, based on this; why is "deploy it in a realistic test fleet" not part of validation? I n…
Is error handling enough? A perfectly valid rule file could hang (but not outright crash) the system, for example.
Re: Preliminary Post Incident Review
#54> How Do We Prevent This From Happening Again? > Software Resiliency and Testing > * Improve Rapid Response Content testing by using testing types such as: > * Local developer testing So no one actually tested the changes before deploying?!
And why is it "local developer testing" and not CI/CD. This makes them look like absolute amateurs.
The fact that they even listed "local developer testing" is pretty weird.
That is just part of the basic process and is hardly the thing that ensures a problem like this doesn't happen.
Re: Preliminary Post Incident Review
#55Earlier quoted context omitted.
> This makes them look like absolute amateurs. This applies also to all Architects and CTO's at all these Fortune 500 companies, who allowed these self updating systems into their critical systems. I would offer a copy of Antifragile to each of these teams: https://en.wikipedia.org/wiki/Antifragile_(book) "Every captain goes down with every ship"
Architects likely do not have a choice. These things are driven by auditors and requirements for things like insurance or PCI and it’s expensive to protest those. I know people who’ve gone full serverless just to lop off the branches of the audit tree about general purpose server operating systems, and now I’m wondering whether anyone is thinking about iOS/ChromeOS for the same reason. The more successful path here i…
> Architects likely do not have a choice.
Architects don't have a choice, CTO are well paid to golf with the CEO and delegate to their teams, Auditors just audit but are not involved with the technical implementations, Developers just develop according to the Spec, and Security team just are a pain in the ass. Nobody owns it...
Everybody get's well paid, and at the end we have to get lessons learned...It's a s*&^&t show...
Re: Preliminary Post Incident Review
#56Earlier quoted context omitted.
> fixing whatever made it possible for "problematic content" to cause "ungraceful" crashes Better not only fix this specific bug but continuously use fuzzing to find more places where external data (including updates) can trigger a crash (or worse RCE)
That is indeed necessary. But it seems to me that putting the interpreter in a place in the OS where it can cause a system crash with the be the behavior that it's allowed to do is a fundamental design choice that is not at all addressed by fuzzing.
An interpreter that handles data downloaded from the internet even.
That's an exploit waiting to happen.
Re: Preliminary Post Incident Review
#57Here is my summary with the marketing bullshit ripped out.
Falcon configuration is shipped with both direct driver updates ("sensor content"), and out of band ("rapid response content"). "Sensor Content" are scripts (*) that ship with the driver. "Rapid response content" are data that can be delivered dynamically.
One way that "Rapid Response Content" is implemented is with templated "Sensor Content" scripts. CrowdStrike can keep the behavior the same but adjust the parameters by shipping "channel" files that fill in the templates.
"Sensor content", including the templates, are a part of the normal test and release process and goes through testing/verification before being signed/shipped. Customers have control over rollouts and testing.
"Rapid Response Content" is deployed through a different channel that customers do not have control over. Crowdstrike shipped a broken channel file that passed validation but was not tested.
They are going to fix this by adding testing of "rapid response" content updates and support the same rollout logic they do for the driver itself.
(*) I'm using the word "script" here loosely. I don't know what these things are, but they sound like scripts.
---
In other words, they have scripts that would crash given garbage arguments. The validator is supposed to check this before they ship, but the validator screwed it up (why is this a part of release and not done at runtime? (!)). It appears they did not test it, they do not do canary deployments or support rollout of these changes, and everything broke.
Corrupting these channel files sounds like a promising way to attack CS, I wonder if anyone is going down that road.
Re: Preliminary Post Incident Review
#58Earlier quoted context omitted.
It’s endemic in the tech security industry - they’ve been mentally colonised by ex-mil and ex-law enforcement (wannabe mil) folks for a long time. I try to use social work terms and principles in professional settings, which blows these people’s minds. Advocacy, capacity evaluation, community engagement, cultural competencies, duty of care, ethics, evidence-based intervention, incentives, macro-, mezzo- and micro-pra…
Even computer security itself is a metaphor (at least in its inception). I often wonder what if instead of using terms like access, key, illegal operation, firewall, etc. we'd instead chosen metaphors from a different domain, for example plumbing. I'm sure a plumbing metaphor could also be found for every computer security concern. Would be so quick to romanticize as well as militarize a field dealing with "leaks," "…
“Fatbergs” expresses some things delivered by some teams very eloquently for me!
Re: Preliminary Post Incident Review
#59A file full of zeros is an "undetected error"?
Good grief.
Re: Preliminary Post Incident Review
#60Earlier quoted context omitted.
>They did write that they intended to fix the bugs I strongly disagree. Add additional validation and enhance error handling say as much as "add band-aids and improve health" in response to a broken arm. Which is not something you'd want to hear from a kindergarten that sends your kid back to you with shattered bones. Note that the things I said were missing are indeed missing in the "mitigation". In particular, addi…
> people were writing that ridiculous SLA's, such as "4 hour response to a vulnerability I didn't see people explaining why this was ridiculous. > make it practically impossible to release well-tested code That falsely presumes the release must be code. CrowdStrike say of the update that caused the crash: "This Rapid Response Content is stored in a proprietary binary file that contains configuration data. It is not c…
>I didn't see people explaining why this was ridiculous.
Because of how it affects priorities and incentives.
E.g.: as of 2024, CrowdStrike didn't implement staggered rollout of Rapid Response content. If you spend a second thinking why that's the case, you'll realize that rapid and staggered are literally antithetical.
>CrowdStrike say of the update that caused the crash: "This Rapid Response Content is stored in a proprietary binary file that contains configuration data. It is not code or a kernel driver."
Well, they are lying.
The data that you feed into an interpreter is code, no matter what they want to call it.