Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on Today's Outage

crowdstrike.com

Post reply on HN
Print view View original

71–80 of 136 posts

Re: Technical Details on Today's Outage

#71

Post by hi-v-rocknroll »

PedroBatista wrote: post #34

Light on technical and light on details. Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.

Yup.

- They don't do enough or the right kind of smoke tests.

- They don't do exponential-canary deployments with an ability to rollback, and instead just YOLO it.

- They don't appear to have a customer-side security / client platform team update approval gating change control process for software updates or for definitions (or whatever they use).

This is fundamentally laziness and/or incompetency.

Re: Technical Details on Today's Outage

#72

Post by notepad0x90 »

calf wrote: post #70
notepad0x90 wrote: post #66

Earlier quoted context omitted.

because the thing that uses them is in kernel mode, and the sensor needs to be performant. at some point, the content must be consumed by the kernel mode sensor. user mode edr's exist but bypassing them is trivial, intercepting syscalls rootkit style and monitoring kernel+usermode memory is the best and most performant way to monitor the whole system.

Apple documentation argues the opposite: "Developers can use frameworks such as DriverKit and NetworkExtension to write USB and human interface drivers, endpoint security tools (like data loss prevention or other endpoint agents), and VPN and network tools, all without needing to write kexts. Third-party security agents should be used only if they take advantage of these APIs or have a robust road map to transition t…

well, this is windows not macos. I don't know what you can do with driverkit for example. maybe microsoft should learn from apple?

Re: Technical Details on Today's Outage

#73

Post by hatsunearu »

Murky3515 wrote: post #15
hatsunearu wrote: post #10

Earlier quoted context omitted.

It sounds like Channel files are just basically definition updates in normal antivirus software; it's not actually code, just some stuff on what the software should "look out for". And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it…

It's not understandable imo. At the very least they should have tests for the loader component that shows it can handle corrupted input. Amateur hour.

Yeah, I re-read my comment and it sounds like I am understanding of them.

But no, saying "channel files aren't kernel code" is just hilarious, considering the channel files define how the actual kernel code is supposed to behave, so it might as well be kernel code. Especially when the bad behavior in question is triggered by bad channel files!!

Re: Technical Details on Today's Outage

#74

Post by timbelina »

Can someone aim me at some RTFM that describes the sensor release and patching process, please? I'm lost trying to understand: When a new version 'n' of the sensor is released, we upgrade a selected batch of machines and do some tests (mostly waiting around :-)) to see that all is well. Then we upgrade the rest of the fleet by OU. However, 'cause we're scaredy cats, we leave some critical kit on n-1 for longer. And some really critical kit even on n-2. (Yeah, there's a risk in not applying patches I know but there are other outage-related risks that we balance; forget that for now) Our assumption is that n-1, n-2, etc are old, stable releases, and so when fan and shit collided yesterday, we just hopped on the console and did a policy update to revert to n-2 and assumed we'd dodged the bullet. But of course, that failed... you know what they say about assumptions :-) So in a long-winded way that leads to my three questions: Why did the 'content update' take out not just n but n-whatever sensors equally as effectively? Are the n-whatever versions not actually stable? And if the n-whatever versions are not actually stable and are being patched, what's the point of the versioning? Cheers!

Re: Technical Details on Today's Outage

#75

Post by timbelina »

hatsunearu wrote: post #10
tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

It sounds like Channel files are just basically definition updates in normal antivirus software; it's not actually code, just some stuff on what the software should "look out for". And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it…

I was reading these two threads:

https://x.com/perpetualmaniac/status/1814376668095754753?s=4...

https://x.com/ananayarora/status/1814269058088304760

The authors explain the coding error and coredump well, but I'm lost: Is the buggy code that they're describing the channel file, or some kernel code that consumes the channel file? Is there a way to tell?

Re: Technical Details on Today's Outage

#77

Post by timbelina »

timbelina wrote: post #75
hatsunearu wrote: post #10

Earlier quoted context omitted.

It sounds like Channel files are just basically definition updates in normal antivirus software; it's not actually code, just some stuff on what the software should "look out for". And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it…

I was reading these two threads: https://x.com/perpetualmaniac/status/1814376668095754753?s=4... https://x.com/ananayarora/status/1814269058088304760 The authors explain the coding error and coredump well, but I'm lost: Is the buggy code that they're describing the channel file, or some kernel code that consumes the channel file? Is there a way to tell?

OK, and another question:-) Can tools like Valgrind and ASan pick up the kinds of errors that are described in those two links from my previous post?

Re: Technical Details on Today's Outage

#78

Post by jefurii »

tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

I have a friend who is a security guard at a bank in Hollywood, CA, who told me the computers at his location started going down between 12:00 and 13:00PDT (19:00-20:00UTC).

I don't understand CrowdStrike's rollout system, but given that people started seeing trouble earlier in the day, surely by that time they could have shut down the servers that were serving the updates, or something??

He also told me that soon after that the street outside the bank (another bank across the street, a hospital several blocks down) was lined with police who started barring entry to the buildings unless people had bank cards. By the time I woke up this morning technical people already knew basically what was going on, but I really underestimated how freaked out the average person must have been today.

Re: Technical Details on Today's Outage

#79

Post by ananayarora »

timbelina wrote: post #77
timbelina wrote: post #75

Earlier quoted context omitted.

I was reading these two threads: https://x.com/perpetualmaniac/status/1814376668095754753?s=4... https://x.com/ananayarora/status/1814269058088304760 The authors explain the coding error and coredump well, but I'm lost: Is the buggy code that they're describing the channel file, or some kernel code that consumes the channel file? Is there a way to tell?

OK, and another question:-) Can tools like Valgrind and ASan pick up the kinds of errors that are described in those two links from my previous post?

Author of the second post here. The first author's stack trace seems to show a fault on csagent.sys which is a bad read on 0x9c. There are some other .sys files loaded up by csagent.sys, and that's where the crash seems to happen, apparently.

As for detection, Zach mentions that modern tooling could've been used to find this, so I'm assuming Valgrind can find this: https://x.com/Perpetualmaniac/status/1814376690958868979

Hope this helps!

Re: Technical Details on Today's Outage

#80

Post by timbelina »

ananayarora wrote: post #79
timbelina wrote: post #77

Earlier quoted context omitted.

OK, and another question:-) Can tools like Valgrind and ASan pick up the kinds of errors that are described in those two links from my previous post?

Author of the second post here. The first author's stack trace seems to show a fault on csagent.sys which is a bad read on 0x9c. There are some other .sys files loaded up by csagent.sys, and that's where the crash seems to happen, apparently. As for detection, Zach mentions that modern tooling could've been used to find this, so I'm assuming Valgrind can find this: https://x.com/Perpetualmaniac/status/181437669095886…

Cheers Ananay!

So if I put this all together:

a) The driver (sensor) csagent.sys includes code that hasn't checked with a tool like Valgrind or ASan or something and so includes some kind of memory management bug.

b) Since n, n-1 and n-2 versions of the sensor all died equally spectacularly, that bug as been around for at least three versions of csagent.sys.

c) The bug can be triggered by getting the csagent.sys to swallow a shitty channel file and since csagent runs in kernel mode, when it crashes it BSOD's the system.

d) Someone at Crowdstrike uploaded a shitty channel file as part of an update process that apparently happens many times a day.

Am I on the right track so far? If so, there's no/inadequate memory management checks in the csagent driver, and either:

1)There were also no checks before the borked channel file was uploaded because of a failure to follow process, or because there was no process, but whatever the case it was an accident.

or

2) Someone uploaded on purpose, not by accident, the borked channel file intending for a nasty outcome (probably not BSOD)

I can't believe that there are not a million checks and balances in place to let (1) happen, but as my grandma used to say, "Don't assume malice where stupidity will do" :-)

Post reply on HN