Preliminary Post Incident Review • Hacker News Board

Re: Preliminary Post Incident Review

#181

Post by pdpi » Thu, Jul 25, 2024, 6:53 AM UTC

Cyphase wrote: post #45

Lots of words about improving testing of the Rapid Response Content, very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes". > Enhance existing error handling in the Content Interpreter. That's it. Also, it sounds like they might have separate "validation" code, based on this; why is "deploy it in a realistic test fleet" not part of validation? I n…

> very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes"

That stood out to me as well.

Their response was the moral equivalent of Apple saying “iTunes crashes when you play a malformed mp3, so here’s how we’re going to improve how we test our mp3s before sending them to you”.

This is a security product that is expected to handle malicious inputs. If they can’t even handle their own inputs without crashing, I don’t like the odds of this thing being itself a potential attack vector.

Re: Preliminary Post Incident Review

#182

Post by aenis » Thu, Jul 25, 2024, 6:56 AM UTC

"Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production."

So they did not test this update at all, even locally. Its going to be interesting how this plays out in courts. The contract they have with us limits their liability significantly, but this - surely - is gross negligence.

Re: Preliminary Post Incident Review

#183

Post by onetokeoverthe » Thu, Jul 25, 2024, 7:07 AM UTC

aenis wrote: post #182

"Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production." So they did not test this update at all, even locally. Its going to be interesting how this plays out in courts. The contract they have with us limits th…

[dead]

Re: Preliminary Post Incident Review

#184

Post by throwaway2037 » Thu, Jul 25, 2024, 7:24 AM UTC

aenis wrote: post #182

"Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production." So they did not test this update at all, even locally. Its going to be interesting how this plays out in courts. The contract they have with us limits th…

As I understand, it is incredibly difficult to prove "gross negligence". It is better to pressure them to settle in a giant class action lawsuit. I am curious what the total amount of settlements / fines will be in the end. I guess ~2B USD.

Re: Preliminary Post Incident Review

#185

Post by FridgeSeal » Thu, Jul 25, 2024, 7:26 AM UTC

notepad0x90 wrote: post #33

coremoff wrote: post #30

Earlier quoted context omitted.

I disagree; it's much longer than it needs to be, is filled with pseudo-technoese to hide that there's little of consequence in there, and the tiny bit of real information in there is couched with distractions and unnecessary detail. As I understand it, they're telling us that the outage was caused by an unspecified bug in the "Content Validator", and that the file that was shipped was done so without testing because…

no, it's one of most well written PIR's I've seen. It establishes terms and procedures after communicating that this isn't an RCA, then they detail the timeline of tests and deployments done and what went wrong. They were not excessively verbose or terse. This is the right way of communicating to the intended audience. It is both technical people, executives and law makers alike that will be reading this. They commun…

If you think this is good, go look at a Cloudflare postmortem. The fly.io ones are good too.

Way less obscure language, way more detail and depth, actually owning the mistakes rather than vaguely waffling on. This write up from CrowdStrike is close to being functionally junk.

Re: Preliminary Post Incident Review

#186

Post by throwaway2037 » Thu, Jul 25, 2024, 7:28 AM UTC

koolba wrote: post #148

squirrel wrote: post #47

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…

> They didn’t allow their clients any opportunity to test those changes before they took effect I’d argue that anyone that agrees to this is the idiot. Sure they have blame for being the source of the problem, but any CXO that signed off on software that a third party can update whenever they’d like is also at fault. It’s not an “if” situation, it’s a “when”.

I felt exactly the same when I read about the outage. What kind of CTO would allow 3rd party "security" software to automatically update? That's just crazy. Of course, your own security team would do some careful (canary-like) upgrades locally... run for a bit... run some tests, then sign-off. Then upgrade in a staged manner.

Re: Preliminary Post Incident Review

#187

Post by novia » Thu, Jul 25, 2024, 7:33 AM UTC

thayne wrote: post #175

So this event is probably close to a worst case scenario for an untested sensor update. But have they never had issues with such untested updates before, like an update resulting in false positives on legitimate software? Because if they did, that should have been a clue that these types if updates should be tested too.

Crowdstrike issues false positives allll the time. They'll fix them and then they'll come back in a future update. One such false positive is an empty file. Crowdstike hates empty files.

Re: Preliminary Post Incident Review

#188

Post by mr_mitm » Thu, Jul 25, 2024, 7:42 AM UTC

toast0 wrote: post #179

mr_mitm wrote: post #93

Earlier quoted context omitted.

Does anyone test their antivirus updates individually as a customer? I thought they happen multiple times a day, who has time for that?

Some sort of comprehensive test is unlikely. But canary / smoke tests, you can do, if the vendor provides the right tools. It's a cycle: pick the latest release, do some small cluster testing, including rollback testing, then roll out to 1%, if those machines are (mostly) still available in 5 minutes, roll out to 2%, if the 3% is (mostly) still available in 5 minutes, roll out to 4%, etc. If updates are fast and ever…

Which EDR vendor provides a mechanism for testing virus signatures? This is the first time I'm hearing it and I'd like to learn more to close that knowledge gap. I always thought they are all updated ASAP, no exceptions.

Re: Preliminary Post Incident Review

#189

Post by gwd » Thu, Jul 25, 2024, 7:48 AM UTC

Cyphase wrote: post #45

Lots of words about improving testing of the Rapid Response Content, very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes". > Enhance existing error handling in the Content Interpreter. That's it. Also, it sounds like they might have separate "validation" code, based on this; why is "deploy it in a realistic test fleet" not part of validation? I n…

Indeed, the very first thing they should be doing is adding fuzzing of their sensor to the test suite, so that it's not possible (or astronomically unlikely) for any corrupt content to crash the system.

Re: Preliminary Post Incident Review

#190

Post by wasyl » Thu, Jul 25, 2024, 7:48 AM UTC

kasabali wrote: post #92

throwaway7ahgb wrote: post #38

Earlier quoted context omitted.

Where do you see that, it looks like there was a bug in the template tester? Or you mean the manual tests?

> Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production.

I don't read it as _bypassing tests_. They have tested the interpreter (`template type`) when it was first released, and they have _validated_ the new template instance (via `content validator`) and assumed this is enough, because it was enough in the past. None of the steps in the usual process were bypassed, and everything was done by the (their) book.

But it looks to me there's no integration test in the process at all. They're effectively unit testing the interpreter (template type), unit testing (validating) the "code" (template instance), but their testing strategy never actually runs the code on the interpreter (or, executes the template instance against the template type).