Live data from Hacker News

CrowdStrike Official RCA is now out [pdf]

crowdstrike.com

21–30 of 39 posts

Re: CrowdStrike Official RCA is now out [pdf]

#21
post #11

A lot of mitigation actions but nothing to really stop it happening again: a fail safe system in their boot start driver. Bad programming and QA caused the issue, but bad design allowed it to happen

famous windows "guru" Alex Ionescu was their main kernel architector for long time, funny he didn't comment anything about that fail

Re: CrowdStrike Official RCA is now out [pdf]

#24
post #3

Is it just me or does it seem like this change simply wasn't tested beyond a simple unit test?

I posit that there are multiple disparate teams involved. -- Team 1 tells Team 2 that the schema is updating. Team 2 updates their schema. Team 2 tests against updated schema (which would be a test file) All green in test. Team 1 doesn't actually follow the schema. Deployment fails.

It’s worse than that. They updated the schema, and tested it with previous data that does not exercise the new parameter. Tests are passing. When they go and actually use the new parameter, it crashes.

The new schema was improperly tested (among a list of other failures).

Re: CrowdStrike Official RCA is now out [pdf]

#25

[flagged]

This problem comes from CrowdStrike's agent kernel subsystem itself, and not Windows. While I agree that Windows, as a client focused operating system, is a hot mess, but I would also compare Windows NT Kernel as much better than Linux Kernel, in terms of code quality and organization. FreeBSD kernel is still my favourite though. At least Windows NT Kernel have a stable ABI, and Rust for Linux have to bindgen the glu…

Windows supports eBPF. CrowdStrike was written before that supports existed, though.

Re: CrowdStrike Official RCA is now out [pdf]

#26
post #22

It doesn't even cover the barest of organisational root cause. How are they planning to do defense in depth and prevent any internal threat actor from wedging every machine in the world?

Crowdstrike takes it self seriously, for a security company. That means don't ask questions of the experts.

Everyone else sees these services as the patsy when the problem happens.

From a technical perspective it's a hot mess (you are spot on). But business says "everything is fine, this is fine, carry on", because it meets their goal of CYA.

Re: CrowdStrike Official RCA is now out [pdf]

#27

Earlier quoted context omitted.

Sounds more like a off by 1 bug that was hidden by regexs if I'm reading correctly

Very easily hidden. Something obtuse like (.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+) or even this (.{4})(.{7})(.{3})(.{6})(.{9})(.{4})(.{7})(.{3})(.{6})(.{9})(.{4})(.{7})(.{3})(.{6})(.{9})(.{4})(.{7})(.{3})(.{6})(.{9})(.{1}) would simply fail to match. And I wouldn't necessarily blame the developer in either scenario - they received a card that says "hey t…

I think that’s just the nature of kernel programming. Once you’re running in kernel space, there are essentially no safety guards, which is why kernel programming is so difficult. Any faults that occur in user space causing a seg fault + core dump do not exist in kernel space. Especially since kernel code generally has to be written in C, it can be quite difficult even for the best engineers to get everything right.

Re: CrowdStrike Official RCA is now out [pdf]

#28

Earlier quoted context omitted.

This problem comes from CrowdStrike's agent kernel subsystem itself, and not Windows. While I agree that Windows, as a client focused operating system, is a hot mess, but I would also compare Windows NT Kernel as much better than Linux Kernel, in terms of code quality and organization. FreeBSD kernel is still my favourite though. At least Windows NT Kernel have a stable ABI, and Rust for Linux have to bindgen the glu…

Windows supports eBPF. CrowdStrike was written before that supports existed, though.

Also eBPF is still in beta for windows and is nowhere near parity with Linux.

Re: CrowdStrike Official RCA is now out [pdf]

#29

kinda sounds like this was a regex bug? > The selection of data in the channel file was done manually and included a regex wildcard matching criterion in the 21st field for all Template Instances, meaning that execution of these tests during development and release builds did not expose the latent out-of-bounds read in the Content Interpreter when provided with 20 rather than 21 inputs.

I don't think so. As far as I understood this, the wildcard match was basically considered a no-op (since anything matches, they probably optimized by not even attempting the match), and so that 21st field was never provided to their Content Interpeter, so it never crashed before. The first time they actually added a non-wildcard match, the Content Interpeter was actually asked to check the 21st field as well, and it crashed because it only had an array of 20 items.

Re: CrowdStrike Official RCA is now out [pdf]

#30

That's a lot of words to say "We did not test a file that gets ingested by a kernel level program, not even once" At no point did they deploy this file to a computer they owned and attempted to boot it. They purposely decided to deploy behavior to every computer they could without even once making sure it wouldn't break from something stupid. Are these people fucking nuts? I do more testing than this and I might be i…

Leeeroy Jenkiiiins!
Post reply on HN