Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preliminary Post Incident Review

crowdstrike.com

Post reply on HN
Print view View original

71–80 of 227 posts

Re: Preliminary Post Incident Review

#71

Post by hello_moto »

squirrel wrote: post #47

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…

> I predict we’ll see other vendors removing similar bonehead “features” very very quietly over the next few months.

Absolutely this is what will happen.

I don't know much about the practice of AV definition-like feature across Cybersecurity but I would imagine there might be a possibility that no vendors do rolling update today because it involves Opt-in/Opt-out which might influence the vendor's speed to identify attack which in turns affect their "Reputation" as well.

"I bought Vendor-A solution but I got hacked and have to pay Ransomware" (with a side note: because I did not consume the latest critical update of AV definition) is what Vendors worried.

Now that this Global Outage happened, it will change the landscape a bit.

Re: Preliminary Post Incident Review

#72

Post by hello_moto »

belter wrote: post #55
acdha wrote: post #48

Earlier quoted context omitted.

Architects likely do not have a choice. These things are driven by auditors and requirements for things like insurance or PCI and it’s expensive to protest those. I know people who’ve gone full serverless just to lop off the branches of the audit tree about general purpose server operating systems, and now I’m wondering whether anyone is thinking about iOS/ChromeOS for the same reason. The more successful path here i…

> Architects likely do not have a choice. Architects don't have a choice, CTO are well paid to golf with the CEO and delegate to their teams, Auditors just audit but are not involved with the technical implementations, Developers just develop according to the Spec, and Security team just are a pain in the ass. Nobody owns it... Everybody get's well paid, and at the end we have to get lessons learned...It's a s*&^&t s…

Seems like everyone thinks that Execs play golf with another Execs to seal the deal regardless how b0rken the system is.

That CTO's job is on the line if the system can't meet the requirement, more so if the system is fucked.

To think that every CTO is dumbass is like saying "everyone is stupid, except me, of course"

Re: Preliminary Post Incident Review

#73

Post by SoftTalker »

Cyphase wrote: post #45

Lots of words about improving testing of the Rapid Response Content, very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes". > Enhance existing error handling in the Content Interpreter. That's it. Also, it sounds like they might have separate "validation" code, based on this; why is "deploy it in a realistic test fleet" not part of validation? I n…

> it sounds like they might have separate "validation" code

That's what stood out to me. From the CS post: "Template Instances are created and configured through the use of the Content Configuration System, which includes the Content Validator that performs validation checks on the content before it is published."

Lesson learned, a "Validator" that is not actually the same program that will be parsing/reading the file in production, is not a complete test. It's not entirely useless, but it doesn't guarantee anything. The production program could have a latent bug that a completely "valid" (by specification) file might trigger.

Re: Preliminary Post Incident Review

#74

Post by mikequinlan »

sgammon wrote: post #70

1) Everything went mostly well 2) The things that did not fail went so great 3) Many many machines did not fail 4) macOS and Linux unaffected 5) Small lil bug in the content verifier 6) Please enjoy this $10 gift card 7) Every windows machine on earth bsod'd but many things worked

Regarding the gift card, TechCrunch says

"On Wednesday, some of the people who posted about the gift card said that when they went to redeem the offer, they got an error message saying the voucher had been canceled. When TechCrunch checked the voucher, the Uber Eats page provided an error message that said the gift card “has been canceled by the issuing party and is no longer valid.”"

https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-ap...

Re: Preliminary Post Incident Review

#75

Post by hello_moto »

duped wrote: post #57

Here is my summary with the marketing bullshit ripped out. Falcon configuration is shipped with both direct driver updates ("sensor content"), and out of band ("rapid response content"). "Sensor Content" are scripts (*) that ship with the driver. "Rapid response content" are data that can be delivered dynamically. One way that "Rapid Response Content" is implemented is with templated "Sensor Content" scripts. CrowdSt…

> Corrupting these channel files sounds like a promising way to attack CS, I wonder if anyone is going down that road.

Would have happened long time ago if it was that easy no?

Re: Preliminary Post Incident Review

#76

Post by hello_moto »

mc32 wrote: post #23
lopkeny12ko wrote: post #11

[flagged]

How can these companies be certified and compliant, etc., and then in practice have horrible SDLC? What was the impact of diverse teams (offshoring)? Often companies don’t have necessary checks to ensure disparateness of teams does not impact quality. Maybe it was zero or maybe it was more.

You're saying there exist a complex software system without a bug despite following best practices to the dot and certified + compliant?

Re: Preliminary Post Incident Review

#77

Post by hello_moto »

coremoff wrote: post #15

Such a disingenuous review; waffle and distraction to hide the important bits (or rather bit: bug in content validator) behind a wall of text that few people are going to finish. If this is how they are going to publish what happened, I don't have any hope that they've actually learned anything from this event. > Throughout this PIR, we have used generalized terminology to describe the Falcon platform for improved re…

In the current situation, it's better to be complete no?

This information is not just for _you_.

Re: Preliminary Post Incident Review

#78

Post by modestygrime »

SoftTalker wrote: post #73
Cyphase wrote: post #45

Lots of words about improving testing of the Rapid Response Content, very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes". > Enhance existing error handling in the Content Interpreter. That's it. Also, it sounds like they might have separate "validation" code, based on this; why is "deploy it in a realistic test fleet" not part of validation? I n…

> it sounds like they might have separate "validation" code That's what stood out to me. From the CS post: "Template Instances are created and configured through the use of the Content Configuration System, which includes the Content Validator that performs validation checks on the content before it is published." Lesson learned, a "Validator" that is not actually the same program that will be parsing/reading the fil…

I'd argue that it is completely useless. They have the actual parser that runs in production and then a separate "test parser" that doesn't actually reflect reality? Why?

Re: Preliminary Post Incident Review

#79

Post by modestygrime »

Narretz wrote: post #40
Ukv wrote: post #28

A summary, to my understanding: * Their software reads config files to determine which behavior to monitor/block * A "problematic" config file made it through automatic validation checks "due to a bug in the Content Validator" * Further testing of the file was skipped because of "trust in the checks performed in the Content Validator" and successful tests of previous versions * The config file causes their software t…

* Further testing of the file was skipped because of "trust in the checks performed in the Content Validator" and successful tests of previous versions that's crazy. How costly can it be to test the file fully in a CI job? I fail to see how this wasn't implemented already.

Just reeks of incompetence. Do they not have e2e smoketests of this stuff?

Re: Preliminary Post Incident Review

#80

Post by mc32 »

hello_moto wrote: post #76
mc32 wrote: post #23

Earlier quoted context omitted.

How can these companies be certified and compliant, etc., and then in practice have horrible SDLC? What was the impact of diverse teams (offshoring)? Often companies don’t have necessary checks to ensure disparateness of teams does not impact quality. Maybe it was zero or maybe it was more.

You're saying there exist a complex software system without a bug despite following best practices to the dot and certified + compliant?

No, but their release process should catch major bugs such as this. After internal QA, you release to small internal dev team, then to select members of other depts willing to dog-food it, then limited external partners then GA? Or something like that so that you have multiple opportunities to catch weird software/hardware interactions before bringing down business critical systems for major and small companies around the planet?
Post reply on HN