Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on Today's Outage

crowdstrike.com

Post reply on HN
Print view View original

101–110 of 136 posts

Re: Technical Details on Today's Outage

#101

Post by RaftPeople »

dalyons wrote: post #91
tail_exchange wrote: post #89

Earlier quoted context omitted.

Another commenter said that this change was a malformed configuration that crashed the application. If this is the case, you wouldn't need days to see this problem manifest, but only a few minutes. If they had rolled it out to 1% of their customers and waited for a couple hours before releasing it everywhere, they probably would have caught it.

A couple of hours is a long time in the world of automated attacks

It only takes a couple of minutes if you first update your on-site set of LIVE systems sitting there to detect a problem.

If problem encountered, don't send it out to everyone else.

Re: Technical Details on Today's Outage

#102

Post by mynameisvlad »

acdha wrote: post #84
userbinator wrote: post #36

Earlier quoted context omitted.

If Windows wasn't as popular, then this might've happened to Linux to macOS instead. Blame CrowdStrike's incompetence, not MS.

It couldn't happen on macOS: Apple stopped letting third-parties run code in the kernel after years of failures like this. It also wouldn't happen on Linux: they use eBPF there which was designed by grownups and validates its inputs.

eBPF exists on Windows, too: https://microsoft.github.io/ebpf-for-windows/

They’re just not using it. They could have not used it for Linux too. The presence of the feature is not enough to guarantee this would’ve never happened in a hypothetical.

Re: Technical Details on Today's Outage

#103

Post by tootie »

grecy wrote: post #42
PedroBatista wrote: post #34

Light on technical and light on details. Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.

You assume the most competent company got the contracts, which is simply not the world we live in. The company that lobbied the hardest and paid the most in bribes got the contracts.

Having been through enough procurement cycles as both a buyer and seller there does not need to be a whit of malfeasance for a bad decision to occur. It's aggressive sales, price wars, poorly informed decision makers, gut instinct, favoritism, familiarity, incumbency, network effects.

You notice how this outage affected hospitals and airlines? There is a strong tendency in software sales for industries to align around one or two leaders. Oh, American chose Crowdstrike? Maybe we at Delta should just do what they did. Or literally Delta hires the VP from American to be their CISO and he just does what he did before.

Vendor selection is hard and buyer's remorse is frequently hard to deal with once you've sunk cost into a migration.

Re: Technical Details on Today's Outage

#104

Post by mattmaroon »

hi-v-rocknroll wrote: post #71
PedroBatista wrote: post #34

Light on technical and light on details. Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.

Yup. - They don't do enough or the right kind of smoke tests. - They don't do exponential-canary deployments with an ability to rollback, and instead just YOLO it. - They don't appear to have a customer-side security / client platform team update approval gating change control process for software updates or for definitions (or whatever they use). This is fundamentally laziness and/or incompetency.

I have been wondering why they didn’t do canary deployments. Seems like quite an obvious thing to do.

Re: Technical Details on Today's Outage

#105

Post by amiga386 »

SAI_Peregrinus wrote: post #19
SoftTalker wrote: post #5

Earlier quoted context omitted.

The kernel driver reads the channel files. It choked on this one, and crashed.

Which implies that any malware capable of replacing these channel files can crash their kernel driver. I wonder if there's a non-crashing way to exploit this & get kernel-space code execution.

Any malware capable of modifying files under C:\Windows\System32 has no need to fiddle with these files because to have that capability means it already got the keys to the kingdom and could wreck the system in a billion different ways.

See "It rather involved being on the other side of this airtight hatchway" https://devblogs.microsoft.com/oldnewthing/20200420-00/?p=10...

Re: Technical Details on Today's Outage

#106

Post by cube00 »

YZF wrote: post #43
chris_nielsen wrote: post #26

Earlier quoted context omitted.

Yeah “how this logic flaw occurred” is the wrong question. How a common bug was rolled out globally with no controls, testing, or rollback strategy is the right question

They're all good questions. The thing that reads the config should have been fuzz tested with something like AFL. Likely should have a lot more tests. Maybe shouldn't run in a device driver. There's almost no doubt there are engineering process and culture issues here. And then absolutely the release process. Rollback is hard I guess once your OS can't boot.

> Rollback is hard I guess once your OS can't boot.

This is why the client needs have enough error handling to realise it's latest update has now caused unsuccessful boot and roll that update back locally to the last known good configuration (or completely back to factory and pull all updates again).

Re: Technical Details on Today's Outage

#107

Post by cyrnel »

kyriakos wrote: post #67

Why is everyone blaming Microsoft? Is this something of an oversight in their side too? Can someone explain?

I'm no kernel expert, but people are saying Microsoft deserves some blame for not exposing necessary functionality to user space, requiring the use of a very-unsafe kernel driver.

Linux provides eBPF and macOS provides system extensions.

I'll also add that Windows itself heavily prioritizes backwards-compatibility over security, which leads companies to seek out third-party solutions for stopping malware instead of design-based mitigations being built into Windows.

Re: Technical Details on Today's Outage

#108

Post by acdha »

mynameisvlad wrote: post #102
acdha wrote: post #84

Earlier quoted context omitted.

It couldn't happen on macOS: Apple stopped letting third-parties run code in the kernel after years of failures like this. It also wouldn't happen on Linux: they use eBPF there which was designed by grownups and validates its inputs.

eBPF exists on Windows, too: https://microsoft.github.io/ebpf-for-windows/ They’re just not using it. They could have not used it for Linux too. The presence of the feature is not enough to guarantee this would’ve never happened in a hypothetical.

No, the fact that they’re actually using eBPF on Linux is what makes it safer. None of this is magic, it’s just a question of following decades of engineering experience.

Similarly, Microsoft clearly sees the benefits but note that they themselves say that’s not production ready yet. I’m certain that this incident will cause people to consider migrating as soon as that changes.

Re: Technical Details on Today's Outage

#109

Post by acdha »

j16sdiz wrote: post #93
acdha wrote: post #84

Earlier quoted context omitted.

It couldn't happen on macOS: Apple stopped letting third-parties run code in the kernel after years of failures like this. It also wouldn't happen on Linux: they use eBPF there which was designed by grownups and validates its inputs.

Kext still exist on macos

Kind of: they’ve been deprecated for 4 years and you have to disable SIP to load them.

https://developer.apple.com/support/kernel-extensions/

They’ve added system extension mechanisms for the most common needs trying to balance the various things people use kexts for against the impact on security, performance, and reliability many kexts had.

Re: Technical Details on Today's Outage

#110

Post by mynameisvlad »

acdha wrote: post #108
mynameisvlad wrote: post #102

Earlier quoted context omitted.

eBPF exists on Windows, too: https://microsoft.github.io/ebpf-for-windows/ They’re just not using it. They could have not used it for Linux too. The presence of the feature is not enough to guarantee this would’ve never happened in a hypothetical.

No, the fact that they’re actually using eBPF on Linux is what makes it safer. None of this is magic, it’s just a question of following decades of engineering experience. Similarly, Microsoft clearly sees the benefits but note that they themselves say that’s not production ready yet. I’m certain that this incident will cause people to consider migrating as soon as that changes.

You’re responding to a hypothetical, not what happened.

Let’s say Linux is the leading OS around the world. How can we be sure that they would actually use eBPF if this was the case?

They would likely choose the fastest option in order to support the platform as quickly as possible. Perhaps eBPF didn’t even exist if they prioritized Linux support and implemented that first, since Falcon was first released in 2013 and eBPF in 2014.

Switching from kernel mode to eBPF would be quite a lift, so if it wasn’t baked in from the start it likely wouldn’t have been added in after the fact.

A decade worth of changes is a lot to confidently say what would have happened. If Linux and MacOS were more popular than Windows, it could have been completely different.

This doesn’t even touch on the massive Debian incident CS had earlier this year, which is not a hypothetical.

Post reply on HN