A lot of mitigation actions but nothing to really stop it happening again: a fail safe system in their boot start driver. Bad programming and QA caused the issue, but bad design allowed it to happen
CrowdStrike Official RCA is now out [pdf]
11–20 of 39 posts
Re: CrowdStrike Official RCA is now out [pdf]
#12[flagged]
[deleted]
Re: CrowdStrike Official RCA is now out [pdf]
#13That's a lot of words to say "We did not test a file that gets ingested by a kernel level program, not even once" At no point did they deploy this file to a computer they owned and attempted to boot it. They purposely decided to deploy behavior to every computer they could without even once making sure it wouldn't break from something stupid. Are these people fucking nuts? I do more testing than this and I might be i…
I think it is worse than that. When I make a change to some code or config, I'll run it locally to make sure that the change has the effect that I want. I know that we are human and that bugs occasionally appear in code. But what I can't understand is that the human who initiated this change decided not to see if it actually did what they wanted it to do.
I've made changes on personal projects that I thought were simple, and yet broke stuff. But CrowdStrike is a multi-billion dollar company -- how can it be possible to have such a broken process. Their RCA document was interesting, but didn't cover any of the interesting issues. It seems that they don't know about the 5 Whys process (https://en.wikipedia.org/wiki/Five_whys) or decided that those answers were so embarrassing that they had to omit them.
Re: CrowdStrike Official RCA is now out [pdf]
#14Is it just me or does it seem like this change simply wasn't tested beyond a simple unit test?
The big thing I was wondering is what their coverage analysis is like. I can see a developer missing this in a hurry, but where’s the review or second-order analysis? Tons of projects with far less importance monitor branch coverage, use fuzz testing and path analysis tools, etc. and while it’s not trivial to test a kernel driver it’s not _that_ hard, especially when you have the resources of a company valued in the tens of billions which allegedly specializes in exactly this kind of work.
The thing I’ve been thinking about are all of the assurances they made about SDLC, testing, secure development practices, etc. They have so many huge customers in regulated industries, government, etc. that they completed almost every certification in existence and seeing this really raises questions about how those assertions were reviewed.
Re: CrowdStrike Official RCA is now out [pdf]
#15[flagged]
This problem comes from CrowdStrike's agent kernel subsystem itself, and not Windows.
While I agree that Windows, as a client focused operating system, is a hot mess, but I would also compare Windows NT Kernel as much better than Linux Kernel, in terms of code quality and organization. FreeBSD kernel is still my favourite though.
At least Windows NT Kernel have a stable ABI, and Rust for Linux have to bindgen the glue layer for each Linux Kernel build.
Re: CrowdStrike Official RCA is now out [pdf]
#16kinda sounds like this was a regex bug? > The selection of data in the channel file was done manually and included a regex wildcard matching criterion in the 21st field for all Template Instances, meaning that execution of these tests during development and release builds did not expose the latent out-of-bounds read in the Content Interpreter when provided with 20 rather than 21 inputs.
Sounds more like a off by 1 bug that was hidden by regexs if I'm reading correctly
Very easily hidden. Something obtuse like
(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)|(.+)
(.{4})(.{7})(.{3})(.{6})(.{9})(.{4})(.{7})(.{3})(.{6})(.{9})(.{4})(.{7})(.{3})(.{6})(.{9})(.{4})(.{7})(.{3})(.{6})(.{9})(.{1})
And I wouldn't necessarily blame the developer in either scenario - they received a card that says "hey the channel file will now have an extra field in it's schema"... noone said "btw it's optional".
Calling it a "first year programming mistake" like I'm reading in some media is somewhat incendiary. I see unmarshalling errors happen all the time.
The forest that we must not miss is the kernel-level driver simply dies with no error recovery and bricks the system.
Re: CrowdStrike Official RCA is now out [pdf]
#17That's a lot of words to say "We did not test a file that gets ingested by a kernel level program, not even once" At no point did they deploy this file to a computer they owned and attempted to boot it. They purposely decided to deploy behavior to every computer they could without even once making sure it wouldn't break from something stupid. Are these people fucking nuts? I do more testing than this and I might be i…
The whole thing smells of silo'ed teams syndrome.
Team 1 tells Team 2 that the schema is updating.
Team 2 updates their schema.
Team 2 tests against updated schema
All green in test.
Team 1 doesn't actually follow the schema.
Deployment fails.
---
It's really hard to assign blame, but I'd put more blame on Team 2 for not being defensive with their inputs enough.
As we all know there are greater issues with their deployment pipelines (lack of canaries, phased rollouts etc.) but no point going over those in this context.
Re: CrowdStrike Official RCA is now out [pdf]
#18That's a lot of words to say "We did not test a file that gets ingested by a kernel level program, not even once" At no point did they deploy this file to a computer they owned and attempted to boot it. They purposely decided to deploy behavior to every computer they could without even once making sure it wouldn't break from something stupid. Are these people fucking nuts? I do more testing than this and I might be i…
I think it is worse than that. When I make a change to some code or config, I'll run it locally to make sure that the change has the effect that I want. I know that we are human and that bugs occasionally appear in code. But what I can't understand is that the human who initiated this change decided not to see if it actually did what they wanted it to do. I've made changes on personal projects that I thought were sim…
> When I make a change to some code or config, I'll run it locally to make sure that the change has the effect that I want.
It's not uncommon for devs to be working against outdated databases / config dumps. Certainly bad practice but when devs have the option of being lazy vs doing chores, they will pick the path of less resistance.
> But what I can't understand is that the human who initiated this change decided not to see if it actually did what they wanted it to do.
We're assuming that the person who changed the code also made the choice to initiate the rollout. They are 2 separate actions which can be made by separate individuals and could also involve many multiple steps in between, each undertaken by a separate individual as well.
Distance from Prod does introduce a sense of malaise and complacency, I've found.
Re: CrowdStrike Official RCA is now out [pdf]
#19Is it just me or does it seem like this change simply wasn't tested beyond a simple unit test?
I posit that there are multiple disparate teams involved.
--
Team 1 tells Team 2 that the schema is updating.
Team 2 updates their schema.
Team 2 tests against updated schema (which would be a test file)
All green in test.
Team 1 doesn't actually follow the schema.
Deployment fails.
Re: CrowdStrike Official RCA is now out [pdf]
#20Well I guess I should post the obligatory
> Some people, when confronted with a problem, think
> “I know, I’ll use regular expressions.”
> Now they have two problems.