121–130 of 227 posts
Earlier quoted context omitted.
Why can't they just do it more like Microsoft security patches, making them mandatory but giving admins control over when they're deployed?
That would be equivalent to asking "would you prefer your fleet to bluescreen now, or later" in this case.
There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…
If indeed this happens, I'd hail this event as a victory overall; but industry experience tells me that most of those companies will say "it'd never happen with us, we're a lot more careful", and keep doing what they're doing.
In this post mortem there are a lot of words but not one of them actually explains what the problem was. which is: what was the process in place and why did it fail?
They also say a "bug in the content validation". Like what kind of bug? Could it have been prevented with proper testing or code review?
Wasn't 'Channel File 291' a garbage file filled with null pointers? Meaning it's problematic content in the same way as filling your parachute bag with ice cream and screws is problematic.
Direct link to the PIR, instead of the list of posts: https://www.crowdstrike.com/blog/falcon-content-update-preli...
Some updates from the hub page:
They published an "executive summary" in PDF format: https://www.crowdstrike.com/wp-content/uploads/2024/07/Crowd...
That includes a couple of bullet points under "Third Party Validation" (independent code/process reviews), which they added to the PIR on the hub page, but not on the dedicated PIR page.
> Updated 2024-07-24 2217 UTC
> ### Third Party Validation
> - Conduct multiple independent third-party security code reviews.
> - Conduct independent reviews of end-to-end quality processes from development through deployment.
>When received by the sensor and loaded into the Content Interpreter, problematic content in Channel File 291 resulted in an out-of-bounds memory read triggering an exception. Wasn't 'Channel File 291' a garbage file filled with null pointers? Meaning it's problematic content in the same way as filling your parachute bag with ice cream and screws is problematic.
>When received by the sensor and loaded into the Content Interpreter, problematic content in Channel File 291 resulted in an out-of-bounds memory read triggering an exception. Wasn't 'Channel File 291' a garbage file filled with null pointers? Meaning it's problematic content in the same way as filling your parachute bag with ice cream and screws is problematic.
They specifically denied that null bytes were the issue in an earlier update. https://www.crowdstrike.com/blog/falcon-update-for-windows-h...
> How Do We Prevent This From Happening Again? > Software Resiliency and Testing > * Improve Rapid Response Content testing by using testing types such as: > * Local developer testing So no one actually tested the changes before deploying?!
And why is it "local developer testing" and not CI/CD. This makes them look like absolute amateurs.
Earlier quoted context omitted.
> How costly can it be to test the file fully in a CI job? It didn't need a CI job. It just needed one person to actually boot and run a Windows instance with the Crowdstrike software installed: a smoke test. TFA is mostly an irrelevent discourse on the product architecture, stuffed with proprietary Crowdstrike jargon, with about a couple of paragraphs dedicated to the actual problem; and they don't mention the non-e…
They mentioned they do dogfooding. Wonder why it did not work for this update.
This was a "content file", and the first time it was interpreted by the kernel driver was when it was pushed to customer production systems worldwide. There was no testing of any sort.
Earlier quoted context omitted.
Why can't they just do it more like Microsoft security patches, making them mandatory but giving admins control over when they're deployed?
That would be equivalent to asking "would you prefer your fleet to bluescreen now, or later" in this case.