Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on Today's Outage

crowdstrike.com

Post reply on HN
Print view View original

41–50 of 136 posts

Re: Technical Details on Today's Outage

#41

Post by omoikane »

hatsunearu wrote: post #10
tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

It sounds like Channel files are just basically definition updates in normal antivirus software; it's not actually code, just some stuff on what the software should "look out for". And it sounds like they shipped some malformed channel file and the software that interprets it can't handle malformed inputs and ate shit. That software happened to be kernel mode, and also marked as boot-critical, so it if falls over, it…

Configuration files should be treated like code and follow the same gradual rollout practices. See also:

https://sre.google/workbook/canarying-releases/

Which starts with "a majority of incidents are triggered by binary or configuration pushes". The stats for config related failures is one link away at

https://sre.google/workbook/postmortem-analysis/

Where it says 31% of outages in 2010-2017 are caused by "configuration push".

Re: Technical Details on Today's Outage

#42

Post by grecy »

PedroBatista wrote: post #34

Light on technical and light on details. Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.

You assume the most competent company got the contracts, which is simply not the world we live in.

The company that lobbied the hardest and paid the most in bribes got the contracts.

Re: Technical Details on Today's Outage

#43

Post by YZF »

chris_nielsen wrote: post #26
patrickthebold wrote: post #22

>The configuration update triggered a logic error that resulted in an operating system crash. > We understand how this issue occurred and we are doing a thorough root cause analysis to determine how this logic flaw occurred. There's always going to be flaws in the logic of the code, the trick is to not have single errors be so catastrophic.

Yeah “how this logic flaw occurred” is the wrong question. How a common bug was rolled out globally with no controls, testing, or rollback strategy is the right question

They're all good questions. The thing that reads the config should have been fuzz tested with something like AFL. Likely should have a lot more tests. Maybe shouldn't run in a device driver. There's almost no doubt there are engineering process and culture issues here.

And then absolutely the release process.

Rollback is hard I guess once your OS can't boot.

Re: Technical Details on Today's Outage

#44

Post by pneumonic »

> we are doing a "root cause analysis to determine how this logic flaw occurred"

That's going to find a cause: a programmer made an error. That's not the root of the problem. The root of the problem is allowing such an error to be released (especially obvious because of its widespread impact).

Re: Technical Details on Today's Outage

#45

Post by geuis »

Weak.

Very weak and over corporate level of ass covering. And it doesn't even come close to doing that.

They should just let the EM of the team involved provide a public detailed response that I'm sure is floating around internally. Just own the problem and address the questions rather than trying to play at politics, quite poorly.

Re: Technical Details on Today's Outage

#46

Post by userbinator »

SkyPuncher wrote: post #21
Analemma_ wrote: post #20

Earlier quoted context omitted.

That’s the danger of running in kernel mode. I’ve seen some people claim this is because the bad file starts a chain of events which concludes in trying to page an unpageable file, which is an application crash in user space but brings down the whole system if it happens in the kernel.

That seems like programming 101 for these systems. In the past, I've worked around this by validating the configuration of a file before attempting to run it. You bail out in a safe way during validation, but still allow a hard error during run time. Doesn't prevent all misconfigured files, but prevents the stuff like.

I think all programmers should have the experience of using and developing on a single-address-space OS with absolutely no protections like DOS, just to encourage them to improve their skills at writing better, actually correct code. When the smallest bugs will crash your system and cause you to lose work, you tend to be a lot more careful with thinking about what your code does instead of just running it to see what happens.

Re: Technical Details on Today's Outage

#48

Post by notepad0x90 »

zug_zug wrote: post #40
notepad0x90 wrote: post #35

Earlier quoted context omitted.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in…

>> You can't take days testing gradual roll outs for this type of content, because that's how long customers are left unprotected by that content. If you can't take days to do it then do a gradual rollout in hours. It's not a high bar.

they reverted it after about one hour. but sure, they didn't need to target all customers all at once, that's a good point.

Re: Technical Details on Today's Outage

#49

Post by Gigachad »

userbinator wrote: post #46
SkyPuncher wrote: post #21

Earlier quoted context omitted.

That seems like programming 101 for these systems. In the past, I've worked around this by validating the configuration of a file before attempting to run it. You bail out in a safe way during validation, but still allow a hard error during run time. Doesn't prevent all misconfigured files, but prevents the stuff like.

I think all programmers should have the experience of using and developing on a single-address-space OS with absolutely no protections like DOS, just to encourage them to improve their skills at writing better, actually correct code. When the smallest bugs will crash your system and cause you to lose work, you tend to be a lot more careful with thinking about what your code does instead of just running it to see what…

Suggesting “Being more careful” never solves these issues because eventually someone somewhere will have a momentary slip up that causes this.

The real takeaway is that we need to design systems so this kind of issue is less possible. Put less code in the kernel, use tools that prevent these kinds of issues, design computers that can roll back the system if they crash.

Re: Technical Details on Today's Outage

#50

Post by lotr5 »

notepad0x90 wrote: post #35
tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in…

they are dumb enough to process their "channel files" in kernel, this should be only done in usermode
Post reply on HN