A lot of mitigation actions but nothing to really stop it happening again: a fail safe system in their boot start driver. Bad programming and QA caused the issue, but bad design allowed it to happen
CrowdStrike Official RCA is now out [pdf]
21–30 of 39 posts
Re: CrowdStrike Official RCA is now out [pdf]
#22Re: CrowdStrike Official RCA is now out [pdf]
#23Re: CrowdStrike Official RCA is now out [pdf]
#24Is it just me or does it seem like this change simply wasn't tested beyond a simple unit test?
I posit that there are multiple disparate teams involved. -- Team 1 tells Team 2 that the schema is updating. Team 2 updates their schema. Team 2 tests against updated schema (which would be a test file) All green in test. Team 1 doesn't actually follow the schema. Deployment fails.
The new schema was improperly tested (among a list of other failures).
Re: CrowdStrike Official RCA is now out [pdf]
#25[flagged]
This problem comes from CrowdStrike's agent kernel subsystem itself, and not Windows. While I agree that Windows, as a client focused operating system, is a hot mess, but I would also compare Windows NT Kernel as much better than Linux Kernel, in terms of code quality and organization. FreeBSD kernel is still my favourite though. At least Windows NT Kernel have a stable ABI, and Rust for Linux have to bindgen the glu…
Re: CrowdStrike Official RCA is now out [pdf]
#26It doesn't even cover the barest of organisational root cause. How are they planning to do defense in depth and prevent any internal threat actor from wedging every machine in the world?
Everyone else sees these services as the patsy when the problem happens.
From a technical perspective it's a hot mess (you are spot on). But business says "everything is fine, this is fine, carry on", because it meets their goal of CYA.
Re: CrowdStrike Official RCA is now out [pdf]
#27Earlier quoted context omitted.
Sounds more like a off by 1 bug that was hidden by regexs if I'm reading correctly
Very easily hidden. Something obtuse like (.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+) or even this (.{4})(.{7})(.{3})(.{6})(.{9})(.{4})(.{7})(.{3})(.{6})(.{9})(.{4})(.{7})(.{3})(.{6})(.{9})(.{4})(.{7})(.{3})(.{6})(.{9})(.{1}) would simply fail to match. And I wouldn't necessarily blame the developer in either scenario - they received a card that says "hey t…
Re: CrowdStrike Official RCA is now out [pdf]
#28Earlier quoted context omitted.
This problem comes from CrowdStrike's agent kernel subsystem itself, and not Windows. While I agree that Windows, as a client focused operating system, is a hot mess, but I would also compare Windows NT Kernel as much better than Linux Kernel, in terms of code quality and organization. FreeBSD kernel is still my favourite though. At least Windows NT Kernel have a stable ABI, and Rust for Linux have to bindgen the glu…
Windows supports eBPF. CrowdStrike was written before that supports existed, though.
Re: CrowdStrike Official RCA is now out [pdf]
#29kinda sounds like this was a regex bug? > The selection of data in the channel file was done manually and included a regex wildcard matching criterion in the 21st field for all Template Instances, meaning that execution of these tests during development and release builds did not expose the latent out-of-bounds read in the Content Interpreter when provided with 20 rather than 21 inputs.
Re: CrowdStrike Official RCA is now out [pdf]
#30That's a lot of words to say "We did not test a file that gets ingested by a kernel level program, not even once" At no point did they deploy this file to a computer they owned and attempted to boot it. They purposely decided to deploy behavior to every computer they could without even once making sure it wouldn't break from something stupid. Are these people fucking nuts? I do more testing than this and I might be i…