Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preliminary Post Incident Review

crowdstrike.com

Post reply on HN
Print view View original

111–120 of 227 posts

Re: Preliminary Post Incident Review

#111

Post by 98codes »

squirrel wrote: post #47

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…

Combined with this, presented as a change they could potentially make, it's a killer:

> Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment.

They weren't doing any test deployments at all before blasting the world with an update? Reckless.

Re: Preliminary Post Incident Review

#112

Post by mikequinlan »

sgammon wrote: post #94
mikequinlan wrote: post #74

Earlier quoted context omitted.

Regarding the gift card, TechCrunch says "On Wednesday, some of the people who posted about the gift card said that when they went to redeem the offer, they got an error message saying the voucher had been canceled. When TechCrunch checked the voucher, the Uber Eats page provided an error message that said the gift card “has been canceled by the issuing party and is no longer valid.”" https://techcrunch.com/2024/07/2…

There's a KB up about this now. To use your voucher, reboot into safe mode and...

On another forum a person replied…

>The system to redeem the card is probably stuck in a boot loop

Re: Preliminary Post Incident Review

#113

Post by xh-dude »

hrpnk wrote: post #109
denton-scratch wrote: post #82

Earlier quoted context omitted.

> How costly can it be to test the file fully in a CI job? It didn't need a CI job. It just needed one person to actually boot and run a Windows instance with the Crowdstrike software installed: a smoke test. TFA is mostly an irrelevent discourse on the product architecture, stuffed with proprietary Crowdstrike jargon, with about a couple of paragraphs dedicated to the actual problem; and they don't mention the non-e…

They mentioned they do dogfooding. Wonder why it did not work for this update.

They discuss dogfooding “Sensor Content”, which isn’t “Rapid Response Content”.

Overall the way this is written up suggests some cultural problems.

Re: Preliminary Post Incident Review

#114

Post by dmazzoni »

SketchySeaBeast wrote: post #97
squirrel wrote: post #47

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…

Unfortunately, putting the onus on risk adverse organizations like hospitals and governments to validate the AV changes means they just won't get pushed and will be chronically exposed. That said, maybe Crowdstrike should considering validating every step of the delivery pipeline before pushing to customers.

Why can't they just do it more like Microsoft security patches, making them mandatory but giving admins control over when they're deployed?

Re: Preliminary Post Incident Review

#115

Post by jiggawatts »

Zironic wrote: post #8
Scaevolus wrote: post #2

"problematic content"? It was a file of all zero bytes. How exactly was that produced?

If I had to guess blindly based on their writeup, it would seem that if their Content Configuration System is given invalid data, instead of aborting the template, it generates a null template. To a degree it makes sense because it's not unusual for a template generator to provide a null response if given invalid inputs however the Content Validator then took that null and published it instead of handling the null ca…

Returning null instead of throwing an exception when an error occurs is the quality of programming I see from junior outsourced developers.

“if (corrupt digital signature) return null;”

is the type of code I see buried in authentication systems, gleefully converting what should be a sudden stop into a shambling zombie of invalid state and null reference exceptions fifty pages of code later in some controller that’s already written to the database on behalf of an attacker.

If I peer into my crystal ball I see a vision of CrowdStrike error handling code quality that looks suspiciously the same.

(If I sound salty, it’s because I’ve been cleaning up their mess since last week.)

Re: Preliminary Post Incident Review

#116

Post by hello_moto »

belter wrote: post #101
hello_moto wrote: post #72

Earlier quoted context omitted.

Seems like everyone thinks that Execs play golf with another Execs to seal the deal regardless how b0rken the system is. That CTO's job is on the line if the system can't meet the requirement, more so if the system is fucked. To think that every CTO is dumbass is like saying "everyone is stupid, except me, of course"

Not all CTO...but you just saw hundreds of companies, who could do better....

That is true, hundred companies have no backup process in place :D

Re: Preliminary Post Incident Review

#117

Post by XlA5vEKsMISoIln »

dmazzoni wrote: post #114
SketchySeaBeast wrote: post #97

Earlier quoted context omitted.

Unfortunately, putting the onus on risk adverse organizations like hospitals and governments to validate the AV changes means they just won't get pushed and will be chronically exposed. That said, maybe Crowdstrike should considering validating every step of the delivery pipeline before pushing to customers.

Why can't they just do it more like Microsoft security patches, making them mandatory but giving admins control over when they're deployed?

That would be equivalent to asking "would you prefer your fleet to bluescreen now, or later" in this case.

Re: Preliminary Post Incident Review

#118

Post by XlA5vEKsMISoIln »

hello_moto wrote: post #71
squirrel wrote: post #47

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…

> I predict we’ll see other vendors removing similar bonehead “features” very very quietly over the next few months. Absolutely this is what will happen. I don't know much about the practice of AV definition-like feature across Cybersecurity but I would imagine there might be a possibility that no vendors do rolling update today because it involves Opt-in/Opt-out which might influence the vendor's speed to identify a…

>Now that this Global Outage happened, it will change the landscape a bit.

I seriously doubt that. Questions like "why should we use CrowdStrike" will be met with "suppose they've learned their lesson".

Re: Preliminary Post Incident Review

#119

Post by sgammon »

rm445 wrote: post #110
sgammon wrote: post #70

1) Everything went mostly well 2) The things that did not fail went so great 3) Many many machines did not fail 4) macOS and Linux unaffected 5) Small lil bug in the content verifier 6) Please enjoy this $10 gift card 7) Every windows machine on earth bsod'd but many things worked

Fun post, but I'll state the obvious because I think many people do believe that every Windows machine BSOD'd. It was only ones with Crowdstrike software. Which is apparently very common but isn't actually pre-installed by Microsoft in Windows, or anything like that. Source: work in a Windows shop and had a normal day.

True, and definitely worth a mention. This is only Microsoft's fault insofar as it was possible at all to crash this way, this broadly, with so little recourse via remote tooling.

Re: Preliminary Post Incident Review

#120

Post by dmazzoni »

EricE wrote: post #59

A file full of zeros is an "undetected error"? Good grief.

It wasn't a file full of zeros that caused the problem.

While some affected users did have a file full of zeros, that was actually a result of the system in the process of trying to download an update, and not the version of the file that caused the crash.

Post reply on HN