Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preliminary Post Incident Review

crowdstrike.com

Post reply on HN
Print view View original

31–40 of 227 posts

Re: Preliminary Post Incident Review

#31

Post by red2awn »

> How Do We Prevent This From Happening Again?

> Software Resiliency and Testing

> * Improve Rapid Response Content testing by using testing types such as:

> * Local developer testing

So no one actually tested the changes before deploying?!

Re: Preliminary Post Incident Review

#33

Post by notepad0x90 »

coremoff wrote: post #30
notepad0x90 wrote: post #26

Earlier quoted context omitted.

> "behind a wall of text that few people are going to finish." heh? it's not that long and very readable.

I disagree; it's much longer than it needs to be, is filled with pseudo-technoese to hide that there's little of consequence in there, and the tiny bit of real information in there is couched with distractions and unnecessary detail. As I understand it, they're telling us that the outage was caused by an unspecified bug in the "Content Validator", and that the file that was shipped was done so without testing because…

no, it's one of most well written PIR's I've seen. It establishes terms and procedures after communicating that this isn't an RCA, then they detail the timeline of tests and deployments done and what went wrong. They were not excessively verbose or terse. This is the right way of communicating to the intended audience. It is both technical people, executives and law makers alike that will be reading this. They communicated their findings clearly without code, screenshots, excessive historical details and other distractions.

Re: Preliminary Post Incident Review

#35

Post by fulafel »

rurban wrote: post #6

They bypassed the tests and staged deployment, because their previous update looked good. Ha. What if they implemented a release process, and follow it? Like everyone else does. Hackers at the workplace, sigh.

Also it must have been a manual testing effort, otherwise there would be no motive to skip it. IOW, missing test automation.

Re: Preliminary Post Incident Review

#37

Post by justusthane »

nodesocket wrote: post #13

Why do they insist on using what sounds like military pseudo jargon throughout the document? ex. sensors? I mean how about hosts, machines, clients?

The sensor isn't a host, machine, or a client. It's the software component that detects threats. I guess maybe you could call it an agent instead, but I think sensor is pretty accepted terminology in the EDR space - it's not specific to Crowdstrike.

Re: Preliminary Post Incident Review

#38

Post by throwaway7ahgb »

rurban wrote: post #6

They bypassed the tests and staged deployment, because their previous update looked good. Ha. What if they implemented a release process, and follow it? Like everyone else does. Hackers at the workplace, sigh.

Where do you see that, it looks like there was a bug in the template tester? Or you mean the manual tests?

Re: Preliminary Post Incident Review

#40

Post by Narretz »

Ukv wrote: post #28

A summary, to my understanding: * Their software reads config files to determine which behavior to monitor/block * A "problematic" config file made it through automatic validation checks "due to a bug in the Content Validator" * Further testing of the file was skipped because of "trust in the checks performed in the Content Validator" and successful tests of previous versions * The config file causes their software t…

* Further testing of the file was skipped because of "trust in the checks performed in the Content Validator" and successful tests of previous versions

that's crazy. How costly can it be to test the file fully in a CI job? I fail to see how this wasn't implemented already.

Post reply on HN