Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on Today's Outage

crowdstrike.com

Post reply on HN
Print view View original

81–90 of 136 posts

Re: Technical Details on Today's Outage

#81

Post by acdha »

andrewinardeer wrote: post #52
vanillax wrote: post #25

Earlier quoted context omitted.

I mean, the whole world was impacted. All they had to do was test this change in a lab with several pcs. Clearly this wasn't a edge case nor a subtle problem. This was clearly a lack of testing.

It was a Friday. Devs just wanted to go home for the weekend.

Leave the spin to the PR people. Their customers pay a great deal of money for 24x7 service, and this wasn’t even a code change but a definition update – a process which should be as well defined and tested as McDonald’s making a hamburger. You wouldn’t excuse getting E. coli from your lunch with “the cook just wanted to go home for the weekend”, and this is a much more expensive service.

Re: Technical Details on Today's Outage

#82

Post by acdha »

notepad0x90 wrote: post #65
lotr5 wrote: post #50

Earlier quoted context omitted.

they are dumb enough to process their "channel files" in kernel, this should be only done in usermode

they need to be processed in kernel mode where the monitoring happens, user mode EDRs are trivial to bypass. they have to be processed by whatever is going to use them, and in this case it is the "lightweight" sensor code in kernel mode.

They need to load data into the kernel eventually but that doesn’t mean that the first time the file is parsed should be in the kernel. For example, on Linux they don’t have this problem because they use the eBPF subsystem and so what’s running in the kernel is validated byte code. Even if they didn’t want to do something that sophisticated they could simply include a validator into the update process, as has been common since the 1980s.

Re: Technical Details on Today's Outage

#83

Post by acdha »

SkyPuncher wrote: post #21
Analemma_ wrote: post #20

Earlier quoted context omitted.

That’s the danger of running in kernel mode. I’ve seen some people claim this is because the bad file starts a chain of events which concludes in trying to page an unpageable file, which is an application crash in user space but brings down the whole system if it happens in the kernel.

That seems like programming 101 for these systems. In the past, I've worked around this by validating the configuration of a file before attempting to run it. You bail out in a safe way during validation, but still allow a hard error during run time. Doesn't prevent all misconfigured files, but prevents the stuff like.

I think it was in the early 90s when I first saw something do A/B style loading where it would record the attempt to load something, recognize that it hadn’t finished, and use the last known good config instead. Anyone studying high-availability systems has a wealth of prior art to learn from.

Re: Technical Details on Today's Outage

#84

Post by acdha »

userbinator wrote: post #36
xyst wrote: post #32

“Technical” detail report reads more like a lawyer generated report. This company is awful. If I ever get a sales pitch from these shit brains, they will get immediately shut down. Also fuck MS and their awful operating system that then spawned this god awful product/company known as “CrowdStike Falcon”

If Windows wasn't as popular, then this might've happened to Linux to macOS instead. Blame CrowdStrike's incompetence, not MS.

It couldn't happen on macOS: Apple stopped letting third-parties run code in the kernel after years of failures like this.

It also wouldn't happen on Linux: they use eBPF there which was designed by grownups and validates its inputs.

Re: Technical Details on Today's Outage

#85

Post by nine_zeros »

hi-v-rocknroll wrote: post #71
PedroBatista wrote: post #34

Light on technical and light on details. Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.

Yup. - They don't do enough or the right kind of smoke tests. - They don't do exponential-canary deployments with an ability to rollback, and instead just YOLO it. - They don't appear to have a customer-side security / client platform team update approval gating change control process for software updates or for definitions (or whatever they use). This is fundamentally laziness and/or incompetency.

> This is fundamentally laziness and/or incompetency.

From management - that couldn't care less about low visibility, low impact projects

Re: Technical Details on Today's Outage

#87

Post by gjm11 »

hatsunearu wrote: post #12
gjm11 wrote: post #8

Earlier quoted context omitted.

It doesn't read to me as trying to dodge anything. They aren't saying "they're not kernel drivers, so everything is OK", they're saying "seeing the .sys on the filenames, you might think they're kernel drivers, but as it happens they're something else". (Maybe there's some subtext that I'm missing, but I don't see how saying "these aren't kernel drivers" makes them look any better, and I do see why they might say it…

> It doesn't read to me as trying to dodge anything. It absolutely reads like this. They are getting blasted online for shipping kernel mode driver updates without proper QA and release engineering. Which just from face value just seems like some insano style engineering. They are saying "it's not actually a kernel mode value" to deflect blame. I mean, I really don't understand why they would make this statement othe…

They're getting blasted for causing a massive worldwide outage due to what is clearly inadequate quality control. I don't see why this is any better if it's "pushed a kernel-mode driver update with bugs in it" than if it's "released a product with buggy kernel-mode stuff that can be made to crash by an innocuous-looking data file, and then pushed a data file that made it crash". Same result either way. Same demonstration of inadequate quality control either way.

I think the story they're telling now, which so far as I know is the truth, looks worse for them, because it requires them to have screwed up their QC twice. Once when they made a product that do such bad things, and once when they pushed the data file to millions of PCs without checking what it did.

So I still don't see how "this particular file happens not to be kernel-mode code" makes them look any better, and therefore I don't see why they'd be saying it "to deflect blame". It doesn't deflect blame; they look just as bad either way.

Re: Technical Details on Today's Outage

#88

Post by rdtsc »

> The update that occurred at 04:09 UTC was designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks

The obvious joke here is CS runs the malicious C2 framework. So the system worked as designed: it prevented further execution and quarantined the affected machines.

But given they say that’s just a configuration file (then why the hell is it suffixed with .sys?), it’s actually plausible. A smart attacker could disguise themselves and use the same facilities as the CS. CS will try to block them and blocks itself in the process?

Re: Technical Details on Today's Outage

#89

Post by tail_exchange »

notepad0x90 wrote: post #35
tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in…

Another commenter said that this change was a malformed configuration that crashed the application. If this is the case, you wouldn't need days to see this problem manifest, but only a few minutes. If they had rolled it out to 1% of their customers and waited for a couple hours before releasing it everywhere, they probably would have caught it.

Re: Technical Details on Today's Outage

#90

Post by nonfamous »

>>> Systems that are not currently impacted will continue to operate as expected, continue to provide protection, and have no risk of experiencing this event in the future.

Given that this incident has now happened twice in the space of months (first on Linux, then on Windows), and that as stated in this very post the root cause analysis is not yet complete, I find that statement of “NO RISK” very hard to believe.

Post reply on HN