Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on Today's Outage

crowdstrike.com

Post reply on HN
Print view View original

21–30 of 136 posts

Re: Technical Details on Today's Outage

#21

Post by SkyPuncher »

Analemma_ wrote: post #20
SkyPuncher wrote: post #17

Earlier quoted context omitted.

My understanding is they basically deployed a configuration file. It seems like these files might be akin to virus signatures or other frequently updated run-time configuration. I actually don't think it's outrageous that these files are rolled out globally, simultaneously. I'm guessing they're updated frequently and _should_ be largely benign. What stands out to me is the fact that a bad config file can crash the sy…

That’s the danger of running in kernel mode. I’ve seen some people claim this is because the bad file starts a chain of events which concludes in trying to page an unpageable file, which is an application crash in user space but brings down the whole system if it happens in the kernel.

That seems like programming 101 for these systems.

In the past, I've worked around this by validating the configuration of a file before attempting to run it. You bail out in a safe way during validation, but still allow a hard error during run time.

Doesn't prevent all misconfigured files, but prevents the stuff like.

Re: Technical Details on Today's Outage

#22

Post by patrickthebold »

>The configuration update triggered a logic error that resulted in an operating system crash.

> We understand how this issue occurred and we are doing a thorough root cause analysis to determine how this logic flaw occurred.

There's always going to be flaws in the logic of the code, the trick is to not have single errors be so catastrophic.

Re: Technical Details on Today's Outage

#23

Post by throwaway346434 »

Murky3515 wrote: post #15
hatsunearu wrote: post #10

Earlier quoted context omitted.

It sounds like Channel files are just basically definition updates in normal antivirus software; it's not actually code, just some stuff on what the software should "look out for". And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it…

It's not understandable imo. At the very least they should have tests for the loader component that shows it can handle corrupted input. Amateur hour.

Halting problem is undecidable.

On the scale of "no one bothered to put error handling or validation in" to "a subtle problem exists for this given input"; you and I lack the information to make a judgement.

Re: Technical Details on Today's Outage

#24

Post by SoftTalker »

Murky3515 wrote: post #15
hatsunearu wrote: post #10

Earlier quoted context omitted.

It sounds like Channel files are just basically definition updates in normal antivirus software; it's not actually code, just some stuff on what the software should "look out for". And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it…

It's not understandable imo. At the very least they should have tests for the loader component that shows it can handle corrupted input. Amateur hour.

And though I don't know, I'm guessing it's not a certainty to say they don't contain "code." It would seem to me that they would have to, otherwise novel attacks that weren't caught by one of their existing algorithms could never be detected.

I'm guessing they contain some combination of pattern/regexp type stuff, and interpreted code/scripting with trigger criteria, etc. that all gets loaded into the "engine" that actually runs the threat detection.

Re: Technical Details on Today's Outage

#25

Post by vanillax »

throwaway346434 wrote: post #23
Murky3515 wrote: post #15

Earlier quoted context omitted.

It's not understandable imo. At the very least they should have tests for the loader component that shows it can handle corrupted input. Amateur hour.

Halting problem is undecidable. On the scale of "no one bothered to put error handling or validation in" to "a subtle problem exists for this given input"; you and I lack the information to make a judgement.

I mean, the whole world was impacted. All they had to do was test this change in a lab with several pcs. Clearly this wasn't a edge case nor a subtle problem. This was clearly a lack of testing.

Re: Technical Details on Today's Outage

#26

Post by chris_nielsen »

patrickthebold wrote: post #22

>The configuration update triggered a logic error that resulted in an operating system crash. > We understand how this issue occurred and we are doing a thorough root cause analysis to determine how this logic flaw occurred. There's always going to be flaws in the logic of the code, the trick is to not have single errors be so catastrophic.

Yeah “how this logic flaw occurred” is the wrong question.

How a common bug was rolled out globally with no controls, testing, or rollback strategy is the right question

Re: Technical Details on Today's Outage

#27

Post by numbsafari »

Murky3515 wrote: post #15
hatsunearu wrote: post #10

Earlier quoted context omitted.

It sounds like Channel files are just basically definition updates in normal antivirus software; it's not actually code, just some stuff on what the software should "look out for". And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it…

It's not understandable imo. At the very least they should have tests for the loader component that shows it can handle corrupted input. Amateur hour.

Agreed. We all know about a really interesting vector for infecting the kernel now. One that is poorly tested, poorly implemented, and poorly secured.

Re: Technical Details on Today's Outage

#28

Post by numbsafari »

SAI_Peregrinus wrote: post #19
SoftTalker wrote: post #5

Earlier quoted context omitted.

The kernel driver reads the channel files. It choked on this one, and crashed.

Which implies that any malware capable of replacing these channel files can crash their kernel driver. I wonder if there's a non-crashing way to exploit this & get kernel-space code execution.

I can guarantee you that you aren't the only one thinking this right now.

Re: Technical Details on Today's Outage

#29

Post by acdha »

throwaway346434 wrote: post #23
Murky3515 wrote: post #15

Earlier quoted context omitted.

It's not understandable imo. At the very least they should have tests for the loader component that shows it can handle corrupted input. Amateur hour.

Halting problem is undecidable. On the scale of "no one bothered to put error handling or validation in" to "a subtle problem exists for this given input"; you and I lack the information to make a judgement.

> you and I lack the information to make a judgement.

Think about this a little harder: what do you know about the number of customers affected? We do actually have enough information to make a judgement - bricking millions of critical systems, a very high percentage of their total Windows customer population, tells us that they don’t have progressive rollouts, don’t fail into a safe mode, and that if they do have tests those tests are catastrophically unlike anything their customers run – all they had to do was launch an EC2 instance and see if it kept running.

Re: Technical Details on Today's Outage

#30

Post by ungreased0675 »

This seems very unsatisfying. Not sure if I was expecting too much, but that’s a lot of words for very little information.

I’d like more information on how these Channel Files are created, tested, and deployed. What’s the minimum number of people that can do it? How fast can the process go?

Post reply on HN