Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on Today's Outage

crowdstrike.com

Post reply on HN
Print view View original

31–40 of 136 posts

Re: Technical Details on Today's Outage

#31

Post by Hizonner »

hatsunearu wrote: post #10
tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

It sounds like Channel files are just basically definition updates in normal antivirus software; it's not actually code, just some stuff on what the software should "look out for". And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it…

> it's not actually code, just some stuff on what the software should "look out for"

If it controls the behavior of a computer, then it's code.

> and it's kind of understandable that channel files might seem safe to update constantly without oversight

Yeah, no, it's not. They pushed an update that crashed the majority of their Windows installed base in a way that couldn't be fixed remotely. It doesn't matter what the update was to. It needed to be tested. There is no way that any deployment pipeline that could fail to catch something that blatant could possibly be "understandable".

... and that kernel mode code shouldn't have been parsing anything with any complexity to begin with. And should have been tested into oblivion, and possibly formally verified.

This is amateur-hour nonsense. Which is what you expect from most of these "Enterprise Cyber Security(TM)" vendors.

... AND the users shouldn't have just gone and shoved that kind of thing into every critical path they could think of.

Re: Technical Details on Today's Outage

#32

Post by xyst »

“Technical” detail report reads more like a lawyer generated report. This company is awful.

If I ever get a sales pitch from these shit brains, they will get immediately shut down.

Also fuck MS and their awful operating system that then spawned this god awful product/company known as “CrowdStike Falcon”

Re: Technical Details on Today's Outage

#33

Post by epcoa »

hatsunearu wrote: post #6
SoftTalker wrote: post #5

Earlier quoted context omitted.

The kernel driver reads the channel files. It choked on this one, and crashed.

That's what I thought. So saying "it's not a kernel mode driver" is technically true, but I don't need to explain why it's a bunch of nonsense to try to damage control their incompetence.

No idea why you’re getting downvoted. A configuration file for code that runs in kernel space is usually effectively kernel code (it certainly was in this case) - obviously there are formal methods to allow kernel code to be configured in a “safe” fashion, but it’s obvious that’s not going on here.

Re: Technical Details on Today's Outage

#35

Post by notepad0x90 »

tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud.

I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation until this one outage.

You can't take days testing gradual roll outs for this type of content, because that's how long customers are left unprotected by that content. Although the root cause is on the channel files, I feel like the driver that processes them should have been able to handle the "logic bug" in question so we'll find out more over time I guess.

For example, with windows defender which runs on virtually all windows systems, the signature updates on billions of devices are pushed immediately (with exception to enterprise systems, but even then there is usually not much testing on signature files themselves, if at all). As far as the devops process Crowdstrike uses to test the channel files, I think it's best to leave commentary on that to actual insiders but these updates happen several times a day sometimes and get pushed to every Crowdstrike customer.

Re: Technical Details on Today's Outage

#36

Post by userbinator »

xyst wrote: post #32

“Technical” detail report reads more like a lawyer generated report. This company is awful. If I ever get a sales pitch from these shit brains, they will get immediately shut down. Also fuck MS and their awful operating system that then spawned this god awful product/company known as “CrowdStike Falcon”

If Windows wasn't as popular, then this might've happened to Linux to macOS instead. Blame CrowdStrike's incompetence, not MS.

Re: Technical Details on Today's Outage

#37

Post by liuliu »

throwaway346434 wrote: post #23
Murky3515 wrote: post #15

Earlier quoted context omitted.

It's not understandable imo. At the very least they should have tests for the loader component that shows it can handle corrupted input. Amateur hour.

Halting problem is undecidable. On the scale of "no one bothered to put error handling or validation in" to "a subtle problem exists for this given input"; you and I lack the information to make a judgement.

Not doing fuzzing on user-input supported feature, especially for AV, is damning.

Re: Technical Details on Today's Outage

#38

Post by automatoney »

To my biased ears it sounds like these configuration-like files are a borderline DSL that maybe isn't being treated as such. I feel like that's a common issue - people assume because you call it a config file, it's not a language, and so it doesn't get treated as actual code that gets interpreted.

Re: Technical Details on Today's Outage

#39

Post by YZF »

SkyPuncher wrote: post #21
Analemma_ wrote: post #20

Earlier quoted context omitted.

That’s the danger of running in kernel mode. I’ve seen some people claim this is because the bad file starts a chain of events which concludes in trying to page an unpageable file, which is an application crash in user space but brings down the whole system if it happens in the kernel.

That seems like programming 101 for these systems. In the past, I've worked around this by validating the configuration of a file before attempting to run it. You bail out in a safe way during validation, but still allow a hard error during run time. Doesn't prevent all misconfigured files, but prevents the stuff like.

Perfect example of where instrumentation guided fuzzing like AFL would almost certainly have found a problem.

I agree with the amateur hour observation. But then most things seem to be.

Re: Technical Details on Today's Outage

#40

Post by zug_zug »

notepad0x90 wrote: post #35
tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in…

>> You can't take days testing gradual roll outs for this type of content, because that's how long customers are left unprotected by that content.

If you can't take days to do it then do a gradual rollout in hours. It's not a high bar.

Post reply on HN