Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike Official RCA is now out [pdf]

crowdstrike.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: CrowdStrike Official RCA is now out [pdf]

#31

Post by simiones »

johnzabroski wrote: post #25
stevefan1999 wrote: post #15

Earlier quoted context omitted.

This problem comes from CrowdStrike's agent kernel subsystem itself, and not Windows. While I agree that Windows, as a client focused operating system, is a hot mess, but I would also compare Windows NT Kernel as much better than Linux Kernel, in terms of code quality and organization. FreeBSD kernel is still my favourite though. At least Windows NT Kernel have a stable ABI, and Rust for Linux have to bindgen the glu…

Windows supports eBPF. CrowdStrike was written before that supports existed, though.

A badly written eBPF program at this level could still prevent you from using your computer though. Not in this specific way, but if you use eBPF to prevent other things from running, and you accidentally deploy an eBPF program that, say, triggers on every process start - then you'll prevent every process from running, and the machine will be just as useless as one that doesn't boot at all.

Re: CrowdStrike Official RCA is now out [pdf]

#32

Post by simiones »

kiririn wrote: post #11

A lot of mitigation actions but nothing to really stop it happening again: a fail safe system in their boot start driver. Bad programming and QA caused the issue, but bad design allowed it to happen

I think the QA issues are by far the most important part. A security component of this type, by its nature, has to be able to prevent your computer from doing anything at all, since any part of userspace (at least) could be compromised.

The "fail safe" for a security component is in fact to prevent any user space code from running at all - better that than having it actively harm other systems, exfiltrate data, destroy connected hardware etc. So, no amount of clever design can prevent the CrowdStrike sensor from nuking your system if bad security rules get deployed.

For example, if a bad definition file makes it think that the legit libc or win32 libraries are compromised, it should prevent any userspace program from running, which is just as destructive as failing during boot.

That is why appropriate QA is critical for this type of program. I would expect any definition update of any kind to be tested on dozens of systems with a wide variety of Windows configurations and known-good software far before ever being deployed to any customer system. It seems that CrowdStrike thought the exact opposite of this, and in fact their customers were the first to ever run their new code end-to-end, not the last...

Re: CrowdStrike Official RCA is now out [pdf]

#33

Post by Ukv »

> In summary, it was the confluence of these issues that resulted in a system crash: [...] the lack of a specific test for non-wildcard matching criteria in the 21st field.

I feel they focus a lot on their content validator lacking a check to catch this specific error (probably since that sounds like a more understandable oversight) when the more glaring issue is that they didn't try actually running this template instance on even a single machine, which would've instantly revealed the issue.

Even for amateur software with no unit/integration tests, the developer will still have typically ran it on their own machine to see it working. Here CrowdStrike seem to have been flying blind, just praying new template instances work if they pass the validation checks.

They do at least promise to "ensure that every new Template Instance is tested" further down.

Re: CrowdStrike Official RCA is now out [pdf]

#34

Post by cptskippy »

studmuffin650 wrote: post #6
portugalportuga wrote: post #4

kinda sounds like this was a regex bug? > The selection of data in the channel file was done manually and included a regex wildcard matching criterion in the 21st field for all Template Instances, meaning that execution of these tests during development and release builds did not expose the latent out-of-bounds read in the Content Interpreter when provided with 20 rather than 21 inputs.

Sounds more like a off by 1 bug that was hidden by regexs if I'm reading correctly

Yeah, my read was that they changed an interface to include an optional parameter but never actually tested the underlying code by providing said optional parameter.

The bug in clients (sensors) wasn't due to regex, the regex was in their integration unit testing which also had a bug and was never supplying the 21st parameter to the client code.

Re: CrowdStrike Official RCA is now out [pdf]

#35

Post by acdha »

simiones wrote: post #32
kiririn wrote: post #11

A lot of mitigation actions but nothing to really stop it happening again: a fail safe system in their boot start driver. Bad programming and QA caused the issue, but bad design allowed it to happen

I think the QA issues are by far the most important part. A security component of this type, by its nature, has to be able to prevent your computer from doing anything at all, since any part of userspace (at least) could be compromised. The "fail safe" for a security component is in fact to prevent any user space code from running at all - better that than having it actively harm other systems, exfiltrate data, destr…

> So, no amount of clever design can prevent the CrowdStrike sensor from nuking your system if bad security rules get deployed.

This is too binary a way to think about a complex system. Availability is also a security goal so we shouldn’t cavalierly trade it for minor risks which are mostly edge cases.

For example, say that the fail-safe was an old, old idea where they kept the second most recent version, and if the system failed to start or crashed repeatedly, it automatically rolled back to the last known good version. That turns this kind of problem into at most a reboot – a huge win every customer would have taken - and the only case it would introduce a vulnerability is if there’s an active attack which only the latest rules will block which is so virulent that the number of systems approximates the number who’ll be affected by a bad update. That’s an unlikely set of events, especially because there’s a really tight window where such a fast-spreading attack wouldn’t have compromised the host before CrowdStrike could ship the update.

Another variation of that idea: any time the system fails to start repeatedly, the service blocks processes other than its updater so normal apps aren’t exposed as potential vectors but the system can self-heal in most cases.

Re: CrowdStrike Official RCA is now out [pdf]

#36

Post by grumple »

Ukv wrote: post #33

> In summary, it was the confluence of these issues that resulted in a system crash: [...] the lack of a specific test for non-wildcard matching criteria in the 21st field. I feel they focus a lot on their content validator lacking a check to catch this specific error (probably since that sounds like a more understandable oversight) when the more glaring issue is that they didn't try actually running this template in…

Absolutely. This is the number one issue I see causing problems with devs on my teams. It is extremely simple to test your damn work. Smoke test it. Make sure the damn machine boots. Make sure the app runs.

This is covered in part by a staged deployment... but that's just having your users test for you. Where's the automated integration test, or just the boot test?

Re: CrowdStrike Official RCA is now out [pdf]

#37

Post by flamedoge »

cptskippy wrote: post #34
studmuffin650 wrote: post #6

Earlier quoted context omitted.

Sounds more like a off by 1 bug that was hidden by regexs if I'm reading correctly

Yeah, my read was that they changed an interface to include an optional parameter but never actually tested the underlying code by providing said optional parameter. The bug in clients (sensors) wasn't due to regex, the regex was in their integration unit testing which also had a bug and was never supplying the 21st parameter to the client code.

regex isn't probably a good thing in a kernel boot code considering it's NP hard

Re: CrowdStrike Official RCA is now out [pdf]

#39

Post by cptskippy »

flamedoge wrote: post #37
cptskippy wrote: post #34

Earlier quoted context omitted.

Yeah, my read was that they changed an interface to include an optional parameter but never actually tested the underlying code by providing said optional parameter. The bug in clients (sensors) wasn't due to regex, the regex was in their integration unit testing which also had a bug and was never supplying the 21st parameter to the client code.

regex isn't probably a good thing in a kernel boot code considering it's NP hard

That's true statement but what does it have to do with the RCA? From what I read it appears the regex was in the integration tests for the template.
Post reply on HN