Live data from Hacker News

Why the CrowdStrike bug hit banks hard

bitsaboutmoney.com

81–90 of 250 posts

Re: Why the CrowdStrike bug hit banks hard

#81
post #2

This is a good writeup, but to be fair it's just not a matter of banking regulations. Basically all big companies are under similar obligations regarding endpoint protection.

Should endpoint protection require kernel level access? At what point does it stop becoming protection and start becoming a liability? Obligatory who watches/protects the watchmen/protector...

> At what point does it stop becoming protection and start becoming a liability?

If such outages were more frequent, then it could definitely become a liability. But such risks have to be balanced against the risk of being compromised and leaking customer data and other confidential trade secrets, and the risk posed by the latter one is far higher, not to say it's also more common.

Re: Why the CrowdStrike bug hit banks hard

#82
post #3

I'm still amazed how the blame shifted from Microsoft to CrowdStrike. Yes, CrowdStrike update caused that -- but applications fail all the time. It was Microsoft's oversight to put it on Windows critical path. And banks/airlines etc were hit hard because their _Windows_ didn't boot, not because of an application crash on a perfectly working Windows.

Microsoft didn't write the Falcon sensor software nor did they put it in the kernel. In fact, Microsoft has been shouting to the heavens trying to shift the blame from CrowdStrike onto the European Commission, because they want people to irrationally hate antitrust so they can turn Windows into shitty iOS and monopolize the security market (and applications market) for it.

Furthermore, Microsoft does actually have some rules regarding what you can and can't put into a signed kernel driver. Specifically, they won't sign kernel code unless they've seen and tested it first. CrowdStrike deliberately circumvented this rule by implementing their own configuration format - really, just a fancy way of loading code into the kernel that Microsoft doesn't have signing control over.

If there is blame to be had here for Microsoft, maybe it's that their kernel code signing program doesn't scrutinize third-party configuration formats hard enough. I mean, if you sign a code loader, you're really signing all possible programs, making code signing irrelevant. And configuration is more often than not, code in a trenchcoat. It's often Turing-complete, and almost certainly more complicated than the actual programming languages used to write the compiled code being signed off on.

But at the same time I imagine Microsoft tried this and got pushback. That might be why they feel (incorrectly) like they can blame the EU for this. Every third-party security solution does absolutely unspeakable things in kernel space that no one with actual computer science training would sign off on, using configuration to wrestle signing control away from Microsoft. Remember: Crowdstrike is designed to backdoor Windows systems so that their owners know if an attack has succeeded, not to make them more secure from attacks in the first place. Corporations are states[0], and states fundamentally suffer from poor legibility: they own and operate far too much stuff for a tribe[1] of humans to meaningfully control or remember.

The problem is that we have two different entities that all have the ability to stop this madness. When states run into this situation, they impose "joint and several liability", which means "I don't care how we precisely assign blame, I'm just going to say you all caused it and move on". In other words, it's Microsoft's fault and it's CrowdStrike's fault.

[0] ancaps fite me

[1] Maximally connected social graph with node degree below Dunbar's number.

Re: Why the CrowdStrike bug hit banks hard

#83
post #58

Maybe the IT departments at the affected orgs take solace in the fact that so many other orgs had issues that the heat is off - but in my opinion this was still a failure of IT itself. There's no reason that update should have been pushed automatically to the entire fleet. If Crowdstrike's software doesn't give you a way to rollout updates on a portion of your network before the entire fleet, it shouldn't be used.

The update bypassed the controls orgs had in place to defer/schedule updates, AFAIK.

If that's the case, that doesn't change GP's point: if Crowdstrike can bypass your org's controls on rolling out updates to its software, it shouldn't be used.

Re: Why the CrowdStrike bug hit banks hard

#85
post #6

Earlier quoted context omitted.

How did Microsoft put it on the Windows critical path? (Informational question—I’m not following the issue super closely, but I thought CrowdStrike was a third-party system. Crowdstrike was wrong to put so much code in the kernel. Microsoft was reportedly legally bound to provide this access and allow third-party code to run in the kernel.)

There was an interesting article that these third parties who lobbied to run in the kernel and microsoft acquiesced about 20 years ago which led us down this path. https://web.archive.org/web/20061023112233/http://software.s...

If you dig a little more about what this is talking about, Microsoft did not actually make any kernel related changes.

This was just Symantec and McAffee ranting about PatchGuard and MS did not remove it.

Re: Why the CrowdStrike bug hit banks hard

#87
post #67
post #41

Earlier quoted context omitted.

Couldn't you just ask some OS APIs provided by something in kernelspace for what you need? In fact, isn't this how macOS does things?

You could, and in fact this is what Microsoft wanted to do. The EU said that they couldn't. And the reason why not is simple. Anything that Microsoft thinks is a good thing to add to the API, they'll add for themselves. When the new API is released, their software is released with it. This gives them a competitive advantage over competitors who have to wait for Microsoft to have the idea that they want, and then scra…

Interesting! I guess there's no way to fix this with further regulation either, since it would be some work to prove MS had access to the API contracts before they released them.

The ultimate lesson then is to stop using MS stuff.

Re: Why the CrowdStrike bug hit banks hard

#88
post #71
post #59

Earlier quoted context omitted.

I guess I don't think that's the sole reason, as I think the incentives would still be in place even if Microsoft authored security software did not run anything in kernel space.

You mean in terms of third-parties wanting that level of access regardless? I agree, but it would be an easy "no" then.

In terms of Microsoft convincing regulators that they aren't and won't use any OS knowledge or private APIs ever.

Re: Why the CrowdStrike bug hit banks hard

#89
post #6
post #3

I'm still amazed how the blame shifted from Microsoft to CrowdStrike. Yes, CrowdStrike update caused that -- but applications fail all the time. It was Microsoft's oversight to put it on Windows critical path. And banks/airlines etc were hit hard because their _Windows_ didn't boot, not because of an application crash on a perfectly working Windows.

How did Microsoft put it on the Windows critical path? (Informational question—I’m not following the issue super closely, but I thought CrowdStrike was a third-party system. Crowdstrike was wrong to put so much code in the kernel. Microsoft was reportedly legally bound to provide this access and allow third-party code to run in the kernel.)

Microsoft added a feature to Windows that allows specially-signed antimalware drivers to be loaded extremely early in the boot sequence and be marked as non-optional. The idea is to give antimalware drivers the opportunity to load first, before anything else has had the chance to start.

Furthermore, if a driver is marked as optional and crashes, Windows can reboot with that optional driver disabled next time, preventing infinite crash/boot loops. Obviously that's no good if your antimalware driver gets disabled, so they can mark theirs as "required." Obviously in the CrowdStrike case, we got the worst of both worlds.

Re: Why the CrowdStrike bug hit banks hard

#90

I like the technical stuff here. I'm not so sure about this: > money is core societal infrastructure, like the power grid and transportation systems are. It would be really bad if hackers working for a foreign government could just turn off money. Sure, it would be inconvenient in the short term. But I think the current design is holding us back. I suspect that most of us would have more to gain than to lose if we ma…

In the short term people would probably starve to death.
Post reply on HN