Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preliminary Post Incident Review

crowdstrike.com

Post reply on HN
Print view View original

151–160 of 227 posts

Re: Preliminary Post Incident Review

#151

Post by WatchDog »

duped wrote: post #57

Here is my summary with the marketing bullshit ripped out. Falcon configuration is shipped with both direct driver updates ("sensor content"), and out of band ("rapid response content"). "Sensor Content" are scripts (*) that ship with the driver. "Rapid response content" are data that can be delivered dynamically. One way that "Rapid Response Content" is implemented is with templated "Sensor Content" scripts. CrowdSt…

> Corrupting these channel files sounds like a promising way to attack CS, I wonder if anyone is going down that road.

You would have to get into the supply chain to do much damage.

Otherwise, you would somehow need access to the hosts running the agent.

If you a threat-actor that already has access to hosts running CS, at a scale that would make the news, why would you blow your access on trying to ruin CS's reputation further?

Perhaps if you are a vendor of a competing or adjacent product that deploys an agent, you could deliberately try and crash the CS agent, but you would be caught.

Re: Preliminary Post Incident Review

#152

Post by mianos »

I see a path to this every day.

An actual scenario: Some developer starts working on pre deployment validation of config files. Let's say in a pipeline.

Most of the time the config files are OK.

Management says: "Why are you spending so long on this project, the sprint plan said one week, we can't approve anything that takes more than a week."

Developer: "This is harder than it looks" (heard that before).

Management: "Well, if the config file is OK then we won't have a problem in production. Stop working on it".

Developer: Stops working on it.

Config file with a syntax error slips through, .. The rest is history

Re: Preliminary Post Incident Review

#153

Post by edm0nd »

com wrote: post #22
nodesocket wrote: post #13

Why do they insist on using what sounds like military pseudo jargon throughout the document? ex. sensors? I mean how about hosts, machines, clients?

It’s endemic in the tech security industry - they’ve been mentally colonised by ex-mil and ex-law enforcement (wannabe mil) folks for a long time. I try to use social work terms and principles in professional settings, which blows these people’s minds. Advocacy, capacity evaluation, community engagement, cultural competencies, duty of care, ethics, evidence-based intervention, incentives, macro-, mezzo- and micro-pra…

"military grade encryption!" aka just AES-256

always makes me laugh

Re: Preliminary Post Incident Review

#154

Post by romwell »

dang wrote: post #65
romwell wrote: post #5

Copying my content from the duplicate thread[1] here: This reads like a bunch of baloney to obscure the real problem. The only relevant part you need to see: >Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data . Problematic content ? Yeah, this is telling exactly nothing. Their mitigation is "ummm we'll test more and maybe not roll th…

> Copying my content from the duplicate thread[1] here Please don't do this! It makes merging threads a pain because then we have to find the duplicate subthreads (i.e. your two comments) and merge the replies as well. Instead, if you or anyone will let us know at hn@ycombinator.com which threads need merging, we can do that. The solution is deduplication, not further duplication!

Oops. Noted!

Apologies for inadvertently adding work!

Somehow, I never realized that duplicate threads were merged (instead of one of them being nuked), because it seems like a lot of work in the first place.

Thanks for doing it!

Re: Preliminary Post Incident Review

#155

Post by romwell »

cratermoon wrote: post #56
romwell wrote: post #12

Earlier quoted context omitted.

That is indeed necessary. But it seems to me that putting the interpreter in a place in the OS where it can cause a system crash with the be the behavior that it's allowed to do is a fundamental design choice that is not at all addressed by fuzzing.

An interpreter that handles data downloaded from the internet even. That's an exploit waiting to happen.

I guess "fight fire with fire" is great adage, so why not fight backdoors with backdoors. What can go wrong.

Re: Preliminary Post Incident Review

#156

Post by yuliyp »

throwanem wrote: post #67
hun3 wrote: post #53

Earlier quoted context omitted.

Is error handling enough? A perfectly valid rule file could hang (but not outright crash) the system, for example.

If the rules are Turing-complete, then sure. I don't see enough in the report to tell one way or another; the way rules are made to sound as if filling templates about equally suggests either (if templates may reference other templates) and there is not a lot more detail. Halting seems relatively easy to manage with something like a watchdog timer, though, compared to a sound, crash- and memory-safe* parser for a who…

No matter what sort of static validation they attempt, they're still risking other unanticipated effects. They could stumble upon a bug in the OS or some driver, they could cause false positives, they could trigger logspew or other excessive resource usage.

Failure can happen in strange ways. When in a position as sensitive as deploying software to far-flung machines in arbitrary environments, they need to be paranoid about those failure modes. Excuses aren't enough.

Re: Preliminary Post Incident Review

#157

Post by yuliyp »

hyperpape wrote: post #128
Narretz wrote: post #42

Earlier quoted context omitted.

And why is it "local developer testing" and not CI/CD. This makes them look like absolute amateurs.

It sure sounds like the "Content Validator" they mention is a form of CI/CD. The problem is that it passed that validation, but was capable of failing in reality.

The content validator is a form of validation done in CI. Their CD pipeline is the bigger problem here: it was extremely reckless given the system it was used in (configuring millions of customer machines in unknown environments). A CD pipeline for a tiny startup's email service can just deploy straight away. Crowdstrike (as they finally realized) need a CD pipeline with much more rigorous validation.

Re: Preliminary Post Incident Review

#158

Post by yuliyp »

anonu wrote: post #123

In my experience with outages, usually the problem lies in some human error not following the process: Someone didn't do something, checks weren't performed, code reviews were skipped, someone got lazy. In this post mortem there are a lot of words but not one of them actually explains what the problem was. which is: what was the process in place and why did it fail? They also say a "bug in the content validation". Li…

> In my experience with outages, usually the problem lies in some human error not following the process

Everyone makes mistakes. Blaming them for making those mistakes doesn't help prevent mistakes in the future.

> what kind of bug? Could it have been prevented with proper testing or code review?

It doesn't matter what the exact details of the bug are. A validator and the thing it tries to defend being imperfect mates is a failure mode. They happened to trip that failure mode spectacularly.

Also saying "proper testing and code review" in a post-mortem is useless like 95% of the time. Short of a culture of rubber-stamping and yolo-merging where there is something to do, it's a truism that any bug could have been detected with a test or caught by a diligent reviewer in code review. But they could also have been (and were) missed. "git gud" is not an incident prevention strategy, it's wishful thinking or blaming the devs unlucky enough to break it.

More useful as follow-ups are things like "this type of failure mode feels very dangerous, we can do something to make those failures impossible or much more likely to be caught"

Re: Preliminary Post Incident Review

#159

Post by yuliyp »

rurban wrote: post #6

They bypassed the tests and staged deployment, because their previous update looked good. Ha. What if they implemented a release process, and follow it? Like everyone else does. Hackers at the workplace, sigh.

This feels natural, though: the first time you do something you do it 10x more slowly because there's a lot more risk. Continuing to do things like that forever isn't realistic. Complacency is a double-edged sword: sometimes it gets us to avoid wasting time and energy on needless worry (the first time someone drives a car they go 5 mph and brake at anything surprising), sometimes it gets us to be too reckless (drivers forgetting to check blind spots or driving at dangerous speeds).

Re: Preliminary Post Incident Review

#160

Post by yuliyp »

romwell wrote: post #3

This reads like a bunch of baloney to obscure the real problem. The only relevant part you need to see: >Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data . Problematic content ? Yeah, this is telling exactly nothing. Their mitigation is "ummm we'll test more and maybe not roll the updates to everyone at once", without any direct exp…

I presume the first two bullet points felt obvious enough to not bother stating: of course you fix the code that crashed. The architectural changes are the more interesting bits, and they're covered reasonably well. Your third point can help but no matter what there's still going to be parts of the interpreter that aren't exercised by the validator because it's not actually running the code. Your fourth one is a fair point: building in watchdogs of some sort to prevent a crashloop would be good. Also having a remote killswitch that can be checked before turning the sensor on would have helped in containing the damage of a crashloop. Your last one I feel like is mostly redundant with a lot of the follow-ups they did commit to.

It's far from perfect (both in terms of the lack of defenses to crashloop in the sensor and in what it said about their previous practices) but calling it a nothing sandwich is a bit hyperbolic.

Post reply on HN