Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on Today's Outage

crowdstrike.com

Post reply on HN
Print view View original

111–120 of 136 posts

Re: Technical Details on Today's Outage

#111

Post by SoftTalker »

amiga386 wrote: post #105
SAI_Peregrinus wrote: post #19

Earlier quoted context omitted.

Which implies that any malware capable of replacing these channel files can crash their kernel driver. I wonder if there's a non-crashing way to exploit this & get kernel-space code execution.

Any malware capable of modifying files under C:\Windows\System32 has no need to fiddle with these files because to have that capability means it already got the keys to the kingdom and could wreck the system in a billion different ways. See "It rather involved being on the other side of this airtight hatchway" https://devblogs.microsoft.com/oldnewthing/20200420-00/?p=10...

And yet the cleanup instructions were for the user to delete a file in that directory. That requires booting into safe mode, but if any random user is able to do that, kiss your systems goodbye, a good social engineer (or disgruntled employee) will own any desktop in your organization if he wants to.

Re: Technical Details on Today's Outage

#112

Post by mardifoufs »

dalyons wrote: post #91
tail_exchange wrote: post #89

Earlier quoted context omitted.

Another commenter said that this change was a malformed configuration that crashed the application. If this is the case, you wouldn't need days to see this problem manifest, but only a few minutes. If they had rolled it out to 1% of their customers and waited for a couple hours before releasing it everywhere, they probably would have caught it.

A couple of hours is a long time in the world of automated attacks

Then they should make their testing pipelines even faster, and make sure that they can go from detecting a new threat->tested definition file as quickly as possible. You genuinely cannot skimp on testing in this case. It's inherent to the update, threat protection and not breaking their consumers systems should be non-negotiable for a release. That means testing before deploying. If they can't do it fast enough, their product is broken.

Re: Technical Details on Today's Outage

#113

Post by mardifoufs »

cyrnel wrote: post #107
kyriakos wrote: post #67

Why is everyone blaming Microsoft? Is this something of an oversight in their side too? Can someone explain?

I'm no kernel expert, but people are saying Microsoft deserves some blame for not exposing necessary functionality to user space, requiring the use of a very-unsafe kernel driver. Linux provides eBPF and macOS provides system extensions. I'll also add that Windows itself heavily prioritizes backwards-compatibility over security, which leads companies to seek out third-party solutions for stopping malware instead of d…

I don't agree. I'm glad Microsoft doesn't provide the functionality to do what crowdstrike does to user space. Crowdstrike acts in a similar way to deeply seated malware, except that it is usually installed voluntarily. But the behavior and capabilities that it has are basically what any malware would dream of, and exposing them to user space would imo create a mess (especially on windows). If anything, this is good as it will make people even more weary of kernel mode software.

And I'm not sure epbf actually allows you to do a lot of the stuff crowdstrike-like software does. I know they use it on Linux though so maybe eBPF has evolved a lot since I last looked at it.

Re: Technical Details on Today's Outage

#114

Post by cyrnel »

mardifoufs wrote: post #113
cyrnel wrote: post #107

Earlier quoted context omitted.

I'm no kernel expert, but people are saying Microsoft deserves some blame for not exposing necessary functionality to user space, requiring the use of a very-unsafe kernel driver. Linux provides eBPF and macOS provides system extensions. I'll also add that Windows itself heavily prioritizes backwards-compatibility over security, which leads companies to seek out third-party solutions for stopping malware instead of d…

I don't agree. I'm glad Microsoft doesn't provide the functionality to do what crowdstrike does to user space. Crowdstrike acts in a similar way to deeply seated malware, except that it is usually installed voluntarily. But the behavior and capabilities that it has are basically what any malware would dream of, and exposing them to user space would imo create a mess (especially on windows). If anything, this is good…

I generally agree with you. It's an either-or thing: either Microsoft secures their OS, or they provide safe ways for users to secure their OS. The first option is a million times better, but having neither option leads us to this mess.

Re: Technical Details on Today's Outage

#115

Post by acdha »

mynameisvlad wrote: post #110
acdha wrote: post #108

Earlier quoted context omitted.

No, the fact that they’re actually using eBPF on Linux is what makes it safer. None of this is magic, it’s just a question of following decades of engineering experience. Similarly, Microsoft clearly sees the benefits but note that they themselves say that’s not production ready yet. I’m certain that this incident will cause people to consider migrating as soon as that changes.

You’re responding to a hypothetical, not what happened. Let’s say Linux is the leading OS around the world. How can we be sure that they would actually use eBPF if this was the case? They would likely choose the fastest option in order to support the platform as quickly as possible. Perhaps eBPF didn’t even exist if they prioritized Linux support and implemented that first, since Falcon was first released in 2013 and…

They are using eBPF right now. That suggests that they, like everyone else, see benefits in using a platform feature when it exists.

Re: Technical Details on Today's Outage

#116

Post by dasil003 »

dalyons wrote: post #91
tail_exchange wrote: post #89

Earlier quoted context omitted.

Another commenter said that this change was a malformed configuration that crashed the application. If this is the case, you wouldn't need days to see this problem manifest, but only a few minutes. If they had rolled it out to 1% of their customers and waited for a couple hours before releasing it everywhere, they probably would have caught it.

A couple of hours is a long time in the world of automated attacks

An automated attack would struggle to reach the level of destruction that this failure had due the scale of Crowdstrike deployment and the direct update vector and kernel mode failure. Even with the most critical type of remote vulnerability it would be difficult to achieve anything approaching this level of damage, and for all we know (and by all probabilities) this update was addressing a much less severe vulnerability.

Re: Technical Details on Today's Outage

#117

Post by amiga386 »

SoftTalker wrote: post #111
amiga386 wrote: post #105

Earlier quoted context omitted.

Any malware capable of modifying files under C:\Windows\System32 has no need to fiddle with these files because to have that capability means it already got the keys to the kingdom and could wreck the system in a billion different ways. See "It rather involved being on the other side of this airtight hatchway" https://devblogs.microsoft.com/oldnewthing/20200420-00/?p=10...

And yet the cleanup instructions were for the user to delete a file in that directory. That requires booting into safe mode, but if any random user is able to do that, kiss your systems goodbye, a good social engineer (or disgruntled employee) will own any desktop in your organization if he wants to.

The point is, malware can't get into that directory without user consent. Having physical access to the machine, rebooting into safe mode and running commands is a stonking big user consent.

I can pwn my own desktop, yes, all I have to do is say "run as administrator". But the point of the security boundary is to make it impossible for software to get these privileges without me actively giving it to them.

If you're shifting the goalposts and imagining the computer does not belong to me, but to an organisation that I'm a mere employee of, they'll be using AD Group Policy to control what I can and can't do, and Bitlocker to encrypt the boot drive. I cannot boot into safe mode without having the tech support department give me a special code to unlock the computer. Again, that's how you get on the other side of the airtight hatch.

Re: Technical Details on Today's Outage

#118

Post by notepad0x90 »

TheOtherHobbes wrote: post #95
notepad0x90 wrote: post #64

Earlier quoted context omitted.

working in software, you should know how insanely complex software is, even google, amazon, microsoft, cloudflare and such have outages. mistakes happen because humans are involved. it is the nature and risk of depending complex systems. bridges by comparison are not that complicated. I actually expected their stock to drop a lot more than this, but goes to show you how valuable they are. investors know that any dip…

In which case "Are you awake and sane?" would be a sensible reality check before heading out. You're trying to hand-wave away the inexcusable. The outage is a symptom. The problem is the lack of even the most basic testing. Clearly these files are sent out without even a minimal sanity check. That is a problem, and it's not something that can be hand-waved away.

In the 3-4 decades of the security industry, testing signature files to see if they trigger a corner case system crash has never been practiced. You and others are proclaiming yourselves to be experts in an area of technology you have no experience in. This was not a software update!!

Re: Technical Details on Today's Outage

#119

Post by notepad0x90 »

Marsymars wrote: post #96
notepad0x90 wrote: post #35

Earlier quoted context omitted.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in…

> They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation until this one outage. They certainly run their software on those many customers' systems, but but based on my experience with them, "protect" isn't a descriptor I'm willing to grant them. We don't have the counter-factual where Crowdstrike doesn't exist,…

Yes, we do have a counter factual, they catch actual APT's they investigated the DNC hack in the 2016 elections and stopped many more attacks. You are utterly clueless in this area to make a comment like that honestly, I don't mean that as an insult but you are talking about a world they don't exist in as if every company has them. Most of their customers get them after getting pwned and learning their lesson the hard way. And availability isn't the only security property their customers desire, keeping information out of threat actors' hands and preventing them from tampering things is also desirable. I really hope you understand that in your hypothetical world without crowdstrike, threat actors still exist.

Re: Technical Details on Today's Outage

#120

Post by dizhn »

bostik wrote: post #100
hatsunearu wrote: post #12

Earlier quoted context omitted.

> It doesn't read to me as trying to dodge anything. It absolutely reads like this. They are getting blasted online for shipping kernel mode driver updates without proper QA and release engineering. Which just from face value just seems like some insano style engineering. They are saying "it's not actually a kernel mode value" to deflect blame. I mean, I really don't understand why they would make this statement othe…

Friend does incident response and Windows forensics, and pointed something (in retrospect) rather obvious out yesterday: the instructions for cleaning up simply told people to "delete .SYS files according to this wildcard". No additional context. That caught his eye, because to him it sounded like madness. Apparently deleting random driver files is a fairly well known way to screw a Windows system up even more than i…

They said to delete a single specific file. Did what you're saying happen before that or something?
Post reply on HN