Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on Today's Outage

crowdstrike.com

Post reply on HN
Print view View original

61–70 of 136 posts

Re: Technical Details on Today's Outage

#61

Post by joshka »

SAI_Peregrinus wrote: post #19
SoftTalker wrote: post #5

Earlier quoted context omitted.

The kernel driver reads the channel files. It choked on this one, and crashed.

Which implies that any malware capable of replacing these channel files can crash their kernel driver. I wonder if there's a non-crashing way to exploit this & get kernel-space code execution.

If you have write access a path like C:\Windows\System32\drivers\CrowdStrike\ (and I'd assume the parent directory), then you pretty much can crash the kernel many ways.

If you have the means to insert an AV config file update in between the config servers and the user's host then you probably can PWN the system pretty easily as well.

What this probably does mean is that Crowdstrike will be receiving some attention from hackers of both hat colors. Here's the bug bounty page ... https://hackerone.com/crowdstrike?type=team

Re: Technical Details on Today's Outage

#62

Post by opello »

SkyPuncher wrote: post #17
tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

My understanding is they basically deployed a configuration file. It seems like these files might be akin to virus signatures or other frequently updated run-time configuration. I actually don't think it's outrageous that these files are rolled out globally, simultaneously. I'm guessing they're updated frequently and _should_ be largely benign. What stands out to me is the fact that a bad config file can crash the sy…

Entertainingly enough I got to see a similar thing happen, where a configuration file was killing hardware in the field. After the failure and remediation multiple CI jobs were put in place (some months later) to do basic validity checks on the files.

The lesson of "multiple parser implementations for the same thing seems bad" and "sanity checks to prevent breaking things are hard heuristics to define" such that further changes were deferred.

All that to say that I can appreciate circumstances in which satisfying "don't crash the system" in response to configuration data can actually be fairly hard to realize. It can very significantly depend on the design of the pieces in question. But I also agree that it's pretty damning.

Re: Technical Details on Today's Outage

#63

Post by joaomacp »

notepad0x90 wrote: post #35
tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in…

> They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation

I certainly don't want to know (through disaster news) about the construction company that built the bridge I drive through everyday, not for another 15 years, not ever!

This kind of software simply should not fail, with such a massive install base on so many sensitive industries. We're better than that, the software industry is starting to mature and there are simple and widely-known procedures that could have been used to prevent it.

I have no idea how CrowdStrike stock has only dropped 10% to the values of 2 months ago. Actually, if the financial troubles you get into are only these, take back what I said, software should be failing a lot (why spend money on robustness when you don't lose money on bugs?)

Re: Technical Details on Today's Outage

#64

Post by notepad0x90 »

joaomacp wrote: post #63
notepad0x90 wrote: post #35

Earlier quoted context omitted.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in…

> They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation I certainly don't want to know (through disaster news) about the construction company that built the bridge I drive through everyday, not for another 15 years, not ever! This kind of software simply should not fail, with such a massive install base on so…

working in software, you should know how insanely complex software is, even google, amazon, microsoft, cloudflare and such have outages. mistakes happen because humans are involved. it is the nature and risk of depending complex systems. bridges by comparison are not that complicated.

I actually expected their stock to drop a lot more than this, but goes to show you how valuable they are. investors know that any dip is only temporary because no one is getting rid of crowdstrike.

Think of the security landscape as early 90's new york city at night and crowdstrike as the big bulky guy with lots of guns who protects you for a fee, if he makes a mistakes and hurts you, you will be mad but in the end your need for protection does not suddenly go away and it was a one time mistake.

Re: Technical Details on Today's Outage

#65

Post by notepad0x90 »

lotr5 wrote: post #50
notepad0x90 wrote: post #35

Earlier quoted context omitted.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in…

they are dumb enough to process their "channel files" in kernel, this should be only done in usermode

they need to be processed in kernel mode where the monitoring happens, user mode EDRs are trivial to bypass. they have to be processed by whatever is going to use them, and in this case it is the "lightweight" sensor code in kernel mode.

Re: Technical Details on Today's Outage

#66

Post by notepad0x90 »

vladvasiliu wrote: post #53
lotr5 wrote: post #50

Earlier quoted context omitted.

they are dumb enough to process their "channel files" in kernel, this should be only done in usermode

While I can understand both arguments for and against a gradual rollout, this is the main issue: why do these things need to be processed in kernel? And if there’s a good reason to do it, why isn’t there some kind of circuit breaker?

because the thing that uses them is in kernel mode, and the sensor needs to be performant. at some point, the content must be consumed by the kernel mode sensor. user mode edr's exist but bypassing them is trivial, intercepting syscalls rootkit style and monitoring kernel+usermode memory is the best and most performant way to monitor the whole system.

Re: Technical Details on Today's Outage

#69

Post by Ekaros »

userbinator wrote: post #16
tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

I'm more surprised at the fact that they didn't appear to have tested it on themselves first. FWIW, at least Microsoft still "dogfoods" (and it's what coined that term), and even if the results of that aren't great, I'm sure they would've caught something of this severity... but then again, maybe not[1]. [1] https://news.ycombinator.com/item?id=18189139

This is what really would concern me too. With this wide spread issue any reasonable testing should have detected it. Having a few dozen machines with different configurations for an few hours should have detected this. This should have been in a smoke test.

Push update to machines, observe, power cycle them, observe...

I could understand error in some rarer setup, but this was so common that it should have been obvious error.

Re: Technical Details on Today's Outage

#70

Post by calf »

notepad0x90 wrote: post #66
vladvasiliu wrote: post #53

Earlier quoted context omitted.

While I can understand both arguments for and against a gradual rollout, this is the main issue: why do these things need to be processed in kernel? And if there’s a good reason to do it, why isn’t there some kind of circuit breaker?

because the thing that uses them is in kernel mode, and the sensor needs to be performant. at some point, the content must be consumed by the kernel mode sensor. user mode edr's exist but bypassing them is trivial, intercepting syscalls rootkit style and monitoring kernel+usermode memory is the best and most performant way to monitor the whole system.

Apple documentation argues the opposite:

"Developers can use frameworks such as DriverKit and NetworkExtension to write USB and human interface drivers, endpoint security tools (like data loss prevention or other endpoint agents), and VPN and network tools, all without needing to write kexts. Third-party security agents should be used only if they take advantage of these APIs or have a robust road map to transition to them and away from kernel extensions."

Specifically the 2nd sentence above says security software should use the APIs, not Apple's kernel extensions.

Post reply on HN