Live data from Hacker News

Preliminary Post Incident Review

crowdstrike.com

131–140 of 227 posts

Re: Preliminary Post Incident Review

#131
post #102

Earlier quoted context omitted.

Some industries are forced by regulation or liability to have something like crowdstrike deployed on their systems. And crowdstrike doesn't have a lot of alternatives that tick as many checkboxes and are as widely recognized.

Please give me an example of that specific regulation.

PCI DSS v4.0 Requirements 5 and 6 speaks very broadly for anti-malware controls, which Crowdstrike provides as EDR, and cybersecurity (liability, ransomware, etc) insurance absolutely requires it from the questionnaires I’ve completed and am required to attest to.

> In its first version, PCI DSS included controls for detecting, removing, blocking, and containing malicious code (malware). Until version 3.2.1, these controls were generically referred to as "anti-virus software", which was incorrect technically because they protect not just against viruses, but also against other known malware variants (worms, trojans, ransomware, spyware, rootkits, adware, backdoors, etc.). As a result, the term "antimalware" is now used not only to refer to viruses, but also to all other types of malicious code, more in line with the requirement's objectives.

> To avoid the ambiguities seen in previous versions of the standard about which operating systems should have an anti-malware solution installed and which should not, a more operational approach has been chosen: the entity should perform a periodic assessment to determine which system components should require an anti-malware solution. All other assets that are determined not to be affected by malware should be included in a list (req. 5.2.3).

> Updates of the anti-malware solution must be performed automatically (req. 5.3.1).

> Finally, the term "real-time scanning" is explicitly included for the anti-malware solution (this is a type of persistent, continuous scanning where a scan for security risks is performed every time a file is received, opened, downloaded, copied or modified). Previously, there was a reference to the fact that anti-malware mechanisms should be actively running, which gave rise to different interpretations.

> Continuous behavioral analysis of systems or processes is incorporated as an accepted anti-malware solution scanning method, as an alternative to traditional periodic (scheduled and on-demand) and real-time (on-access) scans (req. 5.3.2).

https://www.advantio.com/blog/analysis-of-pci-dss-v4.0-part-...

Re: Preliminary Post Incident Review

#132

Earlier quoted context omitted.

They specifically denied that null bytes were the issue in an earlier update. https://www.crowdstrike.com/blog/falcon-update-for-windows-h...

Null pointers, not a null array

I'm not sure what you're saying, but note that a file fundamentally cannot contain a null pointer, a file can just contain various bytes.

Re: Preliminary Post Incident Review

#133
post #53
post #45

Lots of words about improving testing of the Rapid Response Content, very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes". > Enhance existing error handling in the Content Interpreter. That's it. Also, it sounds like they might have separate "validation" code, based on this; why is "deploy it in a realistic test fleet" not part of validation? I n…

Is error handling enough? A perfectly valid rule file could hang (but not outright crash) the system, for example.

Increase counter when you start loading

Have timeout

Decrement counter after successful load and parse

Check counter on startup. If it is like 3, maybe consider you are crashing

Re: Preliminary Post Incident Review

#134
post #70

1) Everything went mostly well 2) The things that did not fail went so great 3) Many many machines did not fail 4) macOS and Linux unaffected 5) Small lil bug in the content verifier 6) Please enjoy this $10 gift card 7) Every windows machine on earth bsod'd but many things worked

I get that canary rollout is tricky in this business, since it's all about stopping the spread of viruses and attacks.

That said, this incident review doesn't mention numbers, unless I missed it; how colossal of a fuck up it was.

The reality is that they don't apologize "bad shit just happens", they work their engineers to the grave, make no apology and completely screw up. This reads like a minor bump in processes.

Crowdstrike engineered the biggest computer attack the world has ever seen, with a sole purpose of preventing those. They're slowly becoming the Oracle of security and I see no sign of improvement here.

Re: Preliminary Post Incident Review

#135
I work on a piece of software that is installed on a very large number of servers we do not own. The crowd strike incident is exactly our nightmare scenario. We are extremely cautious about updates, we roll it out very slowly with tons of metrics and automatic rollbacks. I’ve told my manager to bookmark articles about the crowdstrike incident and share it with anyone who complains about how slow the update process is.

The two golden rules are to let host owners control when to update whenever possible, and when it isn’t to deploy very very slowly. If a customer has a CI/CD system, you should make it possible for them to deploy your updates through the same mechanism. So your change gets all the same deployment safety guardrails and automated tests and rollbacks for free. When that isn’t possible, deploy very slowly and monitor. If you start seeing disruptions in metrics (like agents suddenly not checking in because of a reboot loop) rollback or at least pause the deployment.

Re: Preliminary Post Incident Review

#136

There’s only one sentence that matters: "Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed." This is where they admit that: 1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effe…

They deployed changes to their software directly to customer production machines

This is part of the premise of EDR software.

Re: Preliminary Post Incident Review

#137
post #102

Earlier quoted context omitted.

Some industries are forced by regulation or liability to have something like crowdstrike deployed on their systems. And crowdstrike doesn't have a lot of alternatives that tick as many checkboxes and are as widely recognized.

Please give me an example of that specific regulation.

There's a whole body of regulation around service providers to the U.S. Government making it an effective requirement to use this stuff, starting with the FedRAMP Authorization Act (https://www.congress.gov/117/bills/hr7776/BILLS-117hr7776enr...).

See also Section 4.2.4 of the FedRAMP Moderate Readiness Assessment Report (RAR) which can be found here: https://www.fedramp.gov/documents-templates/ as an example.

You cannot obtain an Authorization To Operate (ATO) unless you've satisfied the Assessor that you're in compliance.

Re: Preliminary Post Incident Review

#138
post #89

Besides missing the actual testing (!), the staged rollout (!), looks like they also weren't fuzzing this kernel driver that routinely takes instant worldwide updates. Oops.

check their developer github, "i write kernel-safe bytecode interpreters" :D, [link redacted]

[dead]

Re: Preliminary Post Incident Review

#139
> How Do We Prevent This From Happening Again?

> * Local developer testing

Yup... now that all machines are internet connected, telemetry has replaced QA departments. There are actual people in positions of power that think that they do not need QA and can just test on customers. If there is anything right in the world, crowdsuck will be destroyed by lawsuits and every decisionmaker involved will never work as such again.

Re: Preliminary Post Incident Review

#140

Earlier quoted context omitted.

> I predict we’ll see other vendors removing similar bonehead “features” very very quietly over the next few months. Absolutely this is what will happen. I don't know much about the practice of AV definition-like feature across Cybersecurity but I would imagine there might be a possibility that no vendors do rolling update today because it involves Opt-in/Opt-out which might influence the vendor's speed to identify a…

>Now that this Global Outage happened, it will change the landscape a bit. I seriously doubt that. Questions like "why should we use CrowdStrike" will be met with "suppose they've learned their lesson".

I'm referring to the landscape how current Cybersecurity vendors deliver "detection definition" (for lack of better phrase) to their customers.

If you don't send them fast to your customer and your customer gets compromised, your reputation gets hit.

If you send them fast, this BSOD happened.

It's more like damn if you do, damn if you don't.

Post reply on HN