Preliminary Post Incident Review • Hacker News Board

Re: Preliminary Post Incident Review

#61

Post by romwell » Wed, Jul 24, 2024, 4:43 PM UTC

GoblinSlayer wrote: post #18

romwell wrote: post #10

Earlier quoted context omitted.

>They did write that they intended to fix the bugs I strongly disagree. Add additional validation and enhance error handling say as much as "add band-aids and improve health" in response to a broken arm. Which is not something you'd want to hear from a kindergarten that sends your kid back to you with shattered bones. Note that the things I said were missing are indeed missing in the "mitigation". In particular, addi…

It's not your kid, so "improve health" is the industry standard response here.

True, but the question is why they can keep getting away with that.

Re: Preliminary Post Incident Review

#62

Post by jvreeland » Wed, Jul 24, 2024, 4:45 PM UTC

I really dislike reading website that take over half the screen and make me read off to the side like this. I can fix it by zooming in but I don't understand why they thought making the navigation take up that much of the screen or not be collapsable was a good move.

Re: Preliminary Post Incident Review

#63

Post by hulitu » Wed, Jul 24, 2024, 4:48 PM UTC

mc32 wrote: post #23

lopkeny12ko wrote: post #11

[flagged]

How can these companies be certified and compliant, etc., and then in practice have horrible SDLC? What was the impact of diverse teams (offshoring)? Often companies don’t have necessary checks to ensure disparateness of teams does not impact quality. Maybe it was zero or maybe it was more.

> How can these companies be certified and compliant, etc., and then in practice have horrible SDLC?

Checklists ?

Re: Preliminary Post Incident Review

#64

Post by mardifoufs » Wed, Jul 24, 2024, 5:07 PM UTC

belter wrote: post #55

acdha wrote: post #48

Earlier quoted context omitted.

Architects likely do not have a choice. These things are driven by auditors and requirements for things like insurance or PCI and it’s expensive to protest those. I know people who’ve gone full serverless just to lop off the branches of the audit tree about general purpose server operating systems, and now I’m wondering whether anyone is thinking about iOS/ChromeOS for the same reason. The more successful path here i…

> Architects likely do not have a choice. Architects don't have a choice, CTO are well paid to golf with the CEO and delegate to their teams, Auditors just audit but are not involved with the technical implementations, Developers just develop according to the Spec, and Security team just are a pain in the ass. Nobody owns it... Everybody get's well paid, and at the end we have to get lessons learned...It's a s*&^&t s…

Some industries are forced by regulation or liability to have something like crowdstrike deployed on their systems. And crowdstrike doesn't have a lot of alternatives that tick as many checkboxes and are as widely recognized.

Re: Preliminary Post Incident Review

#65

Post by dang » Wed, Jul 24, 2024, 5:21 PM UTC

romwell wrote: post #5

Copying my content from the duplicate thread[1] here: This reads like a bunch of baloney to obscure the real problem. The only relevant part you need to see: >Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data . Problematic content ? Yeah, this is telling exactly nothing. Their mitigation is "ummm we'll test more and maybe not roll th…

> Copying my content from the duplicate thread[1] here

Please don't do this! It makes merging threads a pain because then we have to find the duplicate subthreads (i.e. your two comments) and merge the replies as well.

Instead, if you or anyone will let us know at hn@ycombinator.com which threads need merging, we can do that. The solution is deduplication, not further duplication!

Re: Preliminary Post Incident Review

#66

Post by hulitu » Wed, Jul 24, 2024, 5:25 PM UTC

CommanderData wrote: post #17

"We didn't properly test our update." Should be the tldr. On threads there's information about CrordStrike slashing QA team numbers, whether that was a factor should be looked at.

They write perfect software. Why should they test it ? /s

Re: Preliminary Post Incident Review

#67

Post by throwanem » Wed, Jul 24, 2024, 5:37 PM UTC

hun3 wrote: post #53

Cyphase wrote: post #45

Lots of words about improving testing of the Rapid Response Content, very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes". > Enhance existing error handling in the Content Interpreter. That's it. Also, it sounds like they might have separate "validation" code, based on this; why is "deploy it in a realistic test fleet" not part of validation? I n…

Is error handling enough? A perfectly valid rule file could hang (but not outright crash) the system, for example.

If the rules are Turing-complete, then sure. I don't see enough in the report to tell one way or another; the way rules are made to sound as if filling templates about equally suggests either (if templates may reference other templates) and there is not a lot more detail. Halting seems relatively easy to manage with something like a watchdog timer, though, compared to a sound, crash- and memory-safe* parser for a whole programming language, especially if that language exists more or less by accident. (Again, no claim; there's not enough available detail.)

I would not want to do any of this directly on metal, where the only safety is what you make for yourself. But that's the line Crowdstrike are in.

* By EDR standards, at least, where "only" one reboot a week forced entirely by memory lost to an unkillable process counts as exceptionally good.

Re: Preliminary Post Incident Review

#68

Post by [deleted] » Wed, Jul 24, 2024, 5:38 PM UTC

red2awn wrote: post #31

> How Do We Prevent This From Happening Again? > Software Resiliency and Testing > * Improve Rapid Response Content testing by using testing types such as: > * Local developer testing So no one actually tested the changes before deploying?!

[deleted]

Re: Preliminary Post Incident Review

#69

Post by ReaLNero » Wed, Jul 24, 2024, 5:38 PM UTC

hun3 wrote: post #53

Cyphase wrote: post #45

Lots of words about improving testing of the Rapid Response Content, very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes". > Enhance existing error handling in the Content Interpreter. That's it. Also, it sounds like they might have separate "validation" code, based on this; why is "deploy it in a realistic test fleet" not part of validation? I n…

Is error handling enough? A perfectly valid rule file could hang (but not outright crash) the system, for example.

Perhaps set a timeout on the operation then? Given this is kernel it's not as easy as userspace, but I'm sure you could request to set a interrupt on a timer.

Re: Preliminary Post Incident Review

#70

Post by sgammon » Wed, Jul 24, 2024, 5:42 PM UTC

1) Everything went mostly well

2) The things that did not fail went so great

3) Many many machines did not fail

4) macOS and Linux unaffected

5) Small lil bug in the content verifier

6) Please enjoy this $10 gift card

7) Every windows machine on earth bsod'd but many things worked