Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preliminary Post Incident Review

crowdstrike.com

Post reply on HN
Print view View original

91–100 of 227 posts

Re: Preliminary Post Incident Review

#91

Post by YZF »

mc32 wrote: post #23
lopkeny12ko wrote: post #11

[flagged]

How can these companies be certified and compliant, etc., and then in practice have horrible SDLC? What was the impact of diverse teams (offshoring)? Often companies don’t have necessary checks to ensure disparateness of teams does not impact quality. Maybe it was zero or maybe it was more.

Standards generally don't mandate specifics and almost certainly nothing specific to SDLC. At least none I've heard of. Things like FIPS and ISO and SOC2 generally prescribe having a certain process, sometimes they can mandate some specifics (e.g. what ciphers for FIPS). Maybe there should be some release process standards that prescribe how this is done but I'm not aware of any. I think part of the problem is the standard bodies don't really know what to prescribe, this sort of has to come from the community. Maybe not unlike the historical development of other engineering professions. Today being compliant with FIPS doesn't really mean you're secure and being SOC2 compliant doesn't really mean customer data is safe etc. It's more some sort of minimal bar in certain areas of practice and process.

Re: Preliminary Post Incident Review

#92

Post by kasabali »

throwaway7ahgb wrote: post #38
rurban wrote: post #6

They bypassed the tests and staged deployment, because their previous update looked good. Ha. What if they implemented a release process, and follow it? Like everyone else does. Hackers at the workplace, sigh.

Where do you see that, it looks like there was a bug in the template tester? Or you mean the manual tests?

> Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production.

Re: Preliminary Post Incident Review

#93

Post by mr_mitm »

squirrel wrote: post #47

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…

Does anyone test their antivirus updates individually as a customer? I thought they happen multiple times a day, who has time for that?

Re: Preliminary Post Incident Review

#94

Post by sgammon »

mikequinlan wrote: post #74
sgammon wrote: post #70

1) Everything went mostly well 2) The things that did not fail went so great 3) Many many machines did not fail 4) macOS and Linux unaffected 5) Small lil bug in the content verifier 6) Please enjoy this $10 gift card 7) Every windows machine on earth bsod'd but many things worked

Regarding the gift card, TechCrunch says "On Wednesday, some of the people who posted about the gift card said that when they went to redeem the offer, they got an error message saying the voucher had been canceled. When TechCrunch checked the voucher, the Uber Eats page provided an error message that said the gift card “has been canceled by the issuing party and is no longer valid.”" https://techcrunch.com/2024/07/2…

There's a KB up about this now. To use your voucher, reboot into safe mode and...

Re: Preliminary Post Incident Review

#95

Post by packetlost »

mr_mitm wrote: post #93
squirrel wrote: post #47

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…

Does anyone test their antivirus updates individually as a customer? I thought they happen multiple times a day, who has time for that?

Yes? Not consumers typically, but many IT departments with certain risk profiles absolutely do.

Re: Preliminary Post Incident Review

#96

Post by packetlost »

squirrel wrote: post #47

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…

I really wish we would get some regulation as a result of this. I know people that almost died due to hospitals being down. It should be absolutely mandatory for users, IT departments, etc. to be able to control when and where updates happen on their infrastructure but *especially* so for critical infrastructure.

Re: Preliminary Post Incident Review

#97

Post by SketchySeaBeast »

squirrel wrote: post #47

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…

Unfortunately, putting the onus on risk adverse organizations like hospitals and governments to validate the AV changes means they just won't get pushed and will be chronically exposed.

That said, maybe Crowdstrike should considering validating every step of the delivery pipeline before pushing to customers.

Re: Preliminary Post Incident Review

#98

Post by mc32 »

YZF wrote: post #91
mc32 wrote: post #23

Earlier quoted context omitted.

How can these companies be certified and compliant, etc., and then in practice have horrible SDLC? What was the impact of diverse teams (offshoring)? Often companies don’t have necessary checks to ensure disparateness of teams does not impact quality. Maybe it was zero or maybe it was more.

Standards generally don't mandate specifics and almost certainly nothing specific to SDLC. At least none I've heard of. Things like FIPS and ISO and SOC2 generally prescribe having a certain process, sometimes they can mandate some specifics (e.g. what ciphers for FIPS). Maybe there should be some release process standards that prescribe how this is done but I'm not aware of any. I think part of the problem is the st…

Sadly, I agree with your take. All it is is a minimum bar. Many who don't have the above are even worse --tho not necessarily, but as a rule probably yes.

Re: Preliminary Post Incident Review

#99

Post by mc32 »

hello_moto wrote: post #85
mc32 wrote: post #80

Earlier quoted context omitted.

No, but their release process should catch major bugs such as this. After internal QA, you release to small internal dev team, then to select members of other depts willing to dog-food it, then limited external partners then GA? Or something like that so that you have multiple opportunities to catch weird software/hardware interactions before bringing down business critical systems for major and small companies aroun…

> After internal QA, you release to small internal dev team, then to select members of other depts willing to dog-food it, then limited external partners then GA What about AV definition update for 0day swimming in the tubes right now?

Sure, those have happened before, but nothing with an impact like last weekend. That's inexcusable. At least definitions can update themselves out of trouble.

Re: Preliminary Post Incident Review

#100

Post by brcmthrowaway »

l00tr wrote: post #89
cataflam wrote: post #87

Besides missing the actual testing (!), the staged rollout (!), looks like they also weren't fuzzing this kernel driver that routinely takes instant worldwide updates. Oops.

check their developer github, "i write kernel-safe bytecode interpreters" :D, [link redacted]

He Codes With Honor(tm)
Post reply on HN