Preliminary Post Incident Review • Hacker News Board

Re: Preliminary Post Incident Review

#161

Post by procflora » Thu, Jul 25, 2024, 3:15 AM UTC

com wrote: post #58

phaedrus wrote: post #52

Earlier quoted context omitted.

Even computer security itself is a metaphor (at least in its inception). I often wonder what if instead of using terms like access, key, illegal operation, firewall, etc. we'd instead chosen metaphors from a different domain, for example plumbing. I'm sure a plumbing metaphor could also be found for every computer security concern. Would be so quick to romanticize as well as militarize a field dealing with "leaks," "…

“Fatbergs” expresses some things delivered by some teams very eloquently for me!

Alternate dimension PR comment: looks flushable to me

Re: Preliminary Post Incident Review

#162

Post by yuliyp » Thu, Jul 25, 2024, 3:25 AM UTC

gostsamo wrote: post #88

Cowards. Why don't you just stand up and admit that you didn't bother testing everything you send to production? Everything else is smoke and the smell of sulfur.

> Why don't you just stand up and admit that you didn't bother testing everything you send to production?

The "What Happened on July 19, 2024?" section combined with the "Rapid Response Content Deployment" make it very clear to anyone reading that that is the case. Similarly, the discussion of the sensor release process in "Sensor Content" and lack of discussion of a release process in the "Rapid Response Content" section solidify the idea that they didn't consider validated rapid response content causing bad behavior as a thing to worry about.

Re: Preliminary Post Incident Review

#163

Post by mdriley » Thu, Jul 25, 2024, 3:46 AM UTC

> Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production.

It compiled, so they shipped it to everyone all at once without ever running it themselves.

They fell short of "works on my machine".

Re: Preliminary Post Incident Review

#164

Post by broknbottle » Thu, Jul 25, 2024, 3:55 AM UTC

stefan_ wrote: post #129

hrpnk wrote: post #109

Earlier quoted context omitted.

They mentioned they do dogfooding. Wonder why it did not work for this update.

You just got tricked by this dishonest article. The whole section that mentions dogfooding is only about actual updates to the kernel driver. This was not a kernel driver update, the entire section is irrelevant. This was a "content file", and the first time it was interpreted by the kernel driver was when it was pushed to customer production systems worldwide. There was no testing of any sort.

All these people claiming they didn’t have canaries. They actually did but people are in denial that they are the canary for crowdstrike lol

Re: Preliminary Post Incident Review

#165

Post by meshko » Thu, Jul 25, 2024, 4:13 AM UTC

I so hate it when people fill these postmortems with marketing speak. Don't they know it is counterproductive?

Re: Preliminary Post Incident Review

#166

Post by cjbprime » Thu, Jul 25, 2024, 4:17 AM UTC

sgammon wrote: post #119

rm445 wrote: post #110

Earlier quoted context omitted.

Fun post, but I'll state the obvious because I think many people do believe that every Windows machine BSOD'd. It was only ones with Crowdstrike software. Which is apparently very common but isn't actually pre-installed by Microsoft in Windows, or anything like that. Source: work in a Windows shop and had a normal day.

True, and definitely worth a mention. This is only Microsoft's fault insofar as it was possible at all to crash this way, this broadly, with so little recourse via remote tooling.

Which is a non-trivial amount of fault, given that Apple disallows the equivalent behavior on macOS.

Re: Preliminary Post Incident Review

#167

Post by cjbprime » Thu, Jul 25, 2024, 4:19 AM UTC

broknbottle wrote: post #164

stefan_ wrote: post #129

Earlier quoted context omitted.

You just got tricked by this dishonest article. The whole section that mentions dogfooding is only about actual updates to the kernel driver. This was not a kernel driver update, the entire section is irrelevant. This was a "content file", and the first time it was interpreted by the kernel driver was when it was pushed to customer production systems worldwide. There was no testing of any sort.

All these people claiming they didn’t have canaries. They actually did but people are in denial that they are the canary for crowdstrike lol

It's worse than that -- if your strategy actually was to use the customer fleet as QA and monitoring, then it probably wouldn't take you an hour and a half to notice that the fleet was exploding and withdraw the update, as it did here. There was simply no QA anywhere.

Re: Preliminary Post Incident Review

#168

Post by cjbprime » Thu, Jul 25, 2024, 4:22 AM UTC

sudosysgen wrote: post #107

hello_moto wrote: post #86

Earlier quoted context omitted.

If it happened, the industry would have known by now. The group behind it will come out to the public.

This would be the kind of vulnerability that would be worth millions of dollars and used for targeted attacks and/or by state actors. It could take years to uncover (like Pegasus, which took 5 years to be discovered) or never be uncovered at all.

Probably not, if you're implying remote code execution -- it was an out of bounds READ operation, not write, causing an immediate crash. Unlikely to be useful for anything other than taking systems offline (which can certainly be useful, but is not RCE).

Re: Preliminary Post Incident Review

#169

Post by skydhash » Thu, Jul 25, 2024, 4:25 AM UTC

hello_moto wrote: post #140

XlA5vEKsMISoIln wrote: post #118

Earlier quoted context omitted.

>Now that this Global Outage happened, it will change the landscape a bit. I seriously doubt that. Questions like "why should we use CrowdStrike" will be met with "suppose they've learned their lesson".

I'm referring to the landscape how current Cybersecurity vendors deliver "detection definition" (for lack of better phrase) to their customers. If you don't send them fast to your customer and your customer gets compromised, your reputation gets hit. If you send them fast, this BSOD happened. It's more like damn if you do, damn if you don't.

> If you don't send them fast to your customer and your customer gets compromised, your reputation gets hit.

> If you send them fast, this BSOD happened.

> It's more like damn if you do, damn if you don't.

What about notifications? If someone has an update policy that disable auto-updates to a critical piece of infrastructure, you can still let him know that there's a critical update is available. Now, he can do follow his own checklist in order to ensure everything goes well.

Re: Preliminary Post Incident Review

#170

Post by yashafromrussia » Thu, Jul 25, 2024, 4:44 AM UTC

Well I'm glad they at least released a public postmortem on the incident. To be honest, I feel naive saying this, but having worked at a bunch of startups my whole life, I expected companies like CrowdStrike to do better than not testing it on their own machines before deploying an update without the ability to roll it back.