Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preliminary Post Incident Review

crowdstrike.com

Post reply on HN
Print view View original

211–220 of 227 posts

Re: Preliminary Post Incident Review

#211

Post by kccqzy »

Cyphase wrote: post #197
kccqzy wrote: post #174

Earlier quoted context omitted.

Maybe they have the same parser in the validator and the real driver, but the vagaries of the C language mean that when undefined behavior is encountered, it may crash or it may work just by chance.

I understand what you're saying. But ~8.5 million machines in 78 minutes isn't a fluke caused by undefined behavior. All signs so far indicate that they would have caught this if they'd had even a modest test fleet. Setting aside the ways they could have prevented it before it reaching that point.

That's besides the point. Of course they need a test fleet. But in the absence of that, there's a very real chance that the existing bug triggered on customer machines but not their validator. This thread is speculating on the reason why their existing validation didn't catch this issue.

Re: Preliminary Post Incident Review

#212

Post by shepherdjerred »

SoftTalker wrote: post #73
Cyphase wrote: post #45

Lots of words about improving testing of the Rapid Response Content, very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes". > Enhance existing error handling in the Content Interpreter. That's it. Also, it sounds like they might have separate "validation" code, based on this; why is "deploy it in a realistic test fleet" not part of validation? I n…

> it sounds like they might have separate "validation" code That's what stood out to me. From the CS post: "Template Instances are created and configured through the use of the Content Configuration System, which includes the Content Validator that performs validation checks on the content before it is published." Lesson learned, a "Validator" that is not actually the same program that will be parsing/reading the fil…

Parse, don't validate

https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

Re: Preliminary Post Incident Review

#213

Post by shepherdjerred »

rurban wrote: post #177
CommanderData wrote: post #25

Earlier quoted context omitted.

They know better obviously, transcending process and bureaucracy.

Same thing happened with Falcon on Debian before. Later they admitted that they didn't test some platforms they were releasing. Never heard of Docker? How can you keep on with such a Q&R manager? He'll cost them billions

Docker wouldn't help with testing kernel modules. You'd need a VM.

Re: Preliminary Post Incident Review

#214

Post by toast0 »

mr_mitm wrote: post #188
toast0 wrote: post #179

Earlier quoted context omitted.

Some sort of comprehensive test is unlikely. But canary / smoke tests, you can do, if the vendor provides the right tools. It's a cycle: pick the latest release, do some small cluster testing, including rollback testing, then roll out to 1%, if those machines are (mostly) still available in 5 minutes, roll out to 2%, if the 3% is (mostly) still available in 5 minutes, roll out to 4%, etc. If updates are fast and ever…

Which EDR vendor provides a mechanism for testing virus signatures? This is the first time I'm hearing it and I'd like to learn more to close that knowledge gap. I always thought they are all updated ASAP, no exceptions.

Microsoft Defender isn't the most sophisticated EDR out there, but you can manage its updates with WSUS. It's been a long time since I've been subject to a corporate imposed EDR or similar, but I seem to recall them pulling updates from a company owned server for bandwidth savings, if nothing else. You can trickle update those with network controls even if the vendor doesn't provide proper tools.

If corporate can't figure out how to manage software updates on their managed systems, the EDR software is the command and control malware the EDR software is supposed to prevent.

Re: Preliminary Post Incident Review

#215

Post by SirMaster »

I feel like for a system that is this widely used and installed in such a critical position that upon a BSOD crash due to a faulting kernel module like this, the system should be able to automatically roll back to try the previous version on subsequent boot(s).

Re: Preliminary Post Incident Review

#216

Post by XlA5vEKsMISoIln »

hello_moto wrote: post #140
XlA5vEKsMISoIln wrote: post #118

Earlier quoted context omitted.

>Now that this Global Outage happened, it will change the landscape a bit. I seriously doubt that. Questions like "why should we use CrowdStrike" will be met with "suppose they've learned their lesson".

I'm referring to the landscape how current Cybersecurity vendors deliver "detection definition" (for lack of better phrase) to their customers. If you don't send them fast to your customer and your customer gets compromised, your reputation gets hit. If you send them fast, this BSOD happened. It's more like damn if you do, damn if you don't.

Okay, but who has more domain knowledge when to deploy? A "security expert" that created the "security product" that operates with root privileges and full telemetry, or IT staff member that looked at said "security expert" value proposition and didn't have issue with it.

Honestly, this reads as a suggestion that even more blame ought to be shifted to the customer.

Re: Preliminary Post Incident Review

#217

Post by sudosysgen »

cjbprime wrote: post #168
sudosysgen wrote: post #107

Earlier quoted context omitted.

This would be the kind of vulnerability that would be worth millions of dollars and used for targeted attacks and/or by state actors. It could take years to uncover (like Pegasus, which took 5 years to be discovered) or never be uncovered at all.

Probably not, if you're implying remote code execution -- it was an out of bounds READ operation, not write, causing an immediate crash. Unlikely to be useful for anything other than taking systems offline (which can certainly be useful, but is not RCE).

It was a read operation during bytecode template initialization, in a driver that reads userland memory. An out of bound read operation to load code in a driver that maps user memory can easily lead to code execution and privilege escalation: if the attacker finds a way to get the out of bound read into memory they control, they could cause the driver to load a manufactured template and inject bytecode.

It's not clear that this specific vulnerability is exploitable, but it's exactly the kind of vulnerability that could be exploited for code execution.

Re: Preliminary Post Incident Review

#218

Post by romwell »

yuliyp wrote: post #160
romwell wrote: post #3

This reads like a bunch of baloney to obscure the real problem. The only relevant part you need to see: >Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data . Problematic content ? Yeah, this is telling exactly nothing. Their mitigation is "ummm we'll test more and maybe not roll the updates to everyone at once", without any direct exp…

I presume the first two bullet points felt obvious enough to not bother stating: of course you fix the code that crashed. The architectural changes are the more interesting bits, and they're covered reasonably well. Your third point can help but no matter what there's still going to be parts of the interpreter that aren't exercised by the validator because it's not actually running the code. Your fourth one is a fair…

>I presume the first two bullet points felt obvious enough to not bother stating: of course you fix the code that crashed.

I was not talking about the code that crashed.

I guess what I wrote was non-obvious enough that it needs an explanation:

— fixing whatever produced "problematic content":

The release doesn't talk about the subsystem that produced the "problematic content". The part that crashed was the interpreter (consumer of the content); the part that generated the "problematic content" might have worked as intended, for all we know.

— fixing whatever made it possible for "problematic content" to cause "ungraceful" crashes:

I am not talking about fixing this particular crash.

I am talking about design choices that allow such crashes in principle.

In this instance, the interpreter seemed to have been reading memory addresses from a configuration file (or something that would be equivalent to doing that). Adding an additional check will fix this bug, but not the fundamental issue that an interpreter should not be doing that.

>The architectural changes are the more interesting bits, and they're covered reasonably well

They are not covered at all. Are we reading the same press release?

>Your third point can help but no matter what there's still going to be parts of the interpreter that aren't exercised by the validator because it's not actually running the code.

Yes, that's the problem I am pointing out: the "validator" and "interpreter" should be the same code. The "validator" can issue commands to a mock operating system instead of doing real API calls, but it should go through the input with the actual interpreter.

In other words, the interpreter should be a part of the validator.

>It's far from perfect (both in terms of the lack of defenses to crashloop in the sensor and in what it said about their previous practices) but calling it a nothing sandwich is a bit hyperbolic.

Sure; that's my subjective assessment. Personally, I am very dissatisfied with their post-mortem. If you are happy with it, that's fair, but you'd need to say more if you want to make a point in addition to "the architectural changes are covered reasonably well".

Like, which specific changes those would be, for starters.

Post reply on HN