Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on Today's Outage

crowdstrike.com

Post reply on HN
Print view View original

91–100 of 136 posts

Re: Technical Details on Today's Outage

#91

Post by dalyons »

tail_exchange wrote: post #89
notepad0x90 wrote: post #35

Earlier quoted context omitted.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in…

Another commenter said that this change was a malformed configuration that crashed the application. If this is the case, you wouldn't need days to see this problem manifest, but only a few minutes. If they had rolled it out to 1% of their customers and waited for a couple hours before releasing it everywhere, they probably would have caught it.

A couple of hours is a long time in the world of automated attacks

Re: Technical Details on Today's Outage

#92

Post by hello_moto »

xyst wrote: post #32

“Technical” detail report reads more like a lawyer generated report. This company is awful. If I ever get a sales pitch from these shit brains, they will get immediately shut down. Also fuck MS and their awful operating system that then spawned this god awful product/company known as “CrowdStike Falcon”

2 things:

1. critical infrastructure around the globe seemed to depend on CrowdStrike

2. "If I ever get a sales pitch from..." suggested you are in an environment that is far from critical infrastructure.

Re: Technical Details on Today's Outage

#93

Post by j16sdiz »

acdha wrote: post #84
userbinator wrote: post #36

Earlier quoted context omitted.

If Windows wasn't as popular, then this might've happened to Linux to macOS instead. Blame CrowdStrike's incompetence, not MS.

It couldn't happen on macOS: Apple stopped letting third-parties run code in the kernel after years of failures like this. It also wouldn't happen on Linux: they use eBPF there which was designed by grownups and validates its inputs.

Kext still exist on macos

Re: Technical Details on Today's Outage

#94

Post by TheOtherHobbes »

dalyons wrote: post #91
tail_exchange wrote: post #89

Earlier quoted context omitted.

Another commenter said that this change was a malformed configuration that crashed the application. If this is the case, you wouldn't need days to see this problem manifest, but only a few minutes. If they had rolled it out to 1% of their customers and waited for a couple hours before releasing it everywhere, they probably would have caught it.

A couple of hours is a long time in the world of automated attacks

Not as long as the weeks it's going to take to undo this.

Re: Technical Details on Today's Outage

#95

Post by TheOtherHobbes »

notepad0x90 wrote: post #64
joaomacp wrote: post #63

Earlier quoted context omitted.

> They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation I certainly don't want to know (through disaster news) about the construction company that built the bridge I drive through everyday, not for another 15 years, not ever! This kind of software simply should not fail, with such a massive install base on so…

working in software, you should know how insanely complex software is, even google, amazon, microsoft, cloudflare and such have outages. mistakes happen because humans are involved. it is the nature and risk of depending complex systems. bridges by comparison are not that complicated. I actually expected their stock to drop a lot more than this, but goes to show you how valuable they are. investors know that any dip…

In which case "Are you awake and sane?" would be a sensible reality check before heading out.

You're trying to hand-wave away the inexcusable. The outage is a symptom. The problem is the lack of even the most basic testing.

Clearly these files are sent out without even a minimal sanity check. That is a problem, and it's not something that can be hand-waved away.

Re: Technical Details on Today's Outage

#96

Post by Marsymars »

notepad0x90 wrote: post #35
tail_exchange wrote: post #3

Can someone who actually understands what CrowdStrike does explain to me why on earth they don't have some kind of gradual rollout for changes? It seems like their updates go out everywhere all at once, and this sounds absolutely insane for a company at this scale.

This "channel file" is equivalent to an AV signature file. Crowdstrike is the company, the product here is "Falcon" which does behavioral monitoring of processes both on the device and using logs collected from the device in the cloud. I can see your perspective, but you should consider this: They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in…

> They protect these many companies, industries and even countries at such a global scale and you haven't even heard of them in the last 15 years of their operation until this one outage.

They certainly run their software on those many customers' systems, but but based on my experience with them, "protect" isn't a descriptor I'm willing to grant them.

We don't have the counter-factual where Crowdstrike doesn't exist, but I'm not convinced that they've been a net economic or security benefit to the world over the span of their existence.

Re: Technical Details on Today's Outage

#97

Post by mynameisvlad »

dalyons wrote: post #91
tail_exchange wrote: post #89

Earlier quoted context omitted.

Another commenter said that this change was a malformed configuration that crashed the application. If this is the case, you wouldn't need days to see this problem manifest, but only a few minutes. If they had rolled it out to 1% of their customers and waited for a couple hours before releasing it everywhere, they probably would have caught it.

A couple of hours is a long time in the world of automated attacks

A couple of hours is absolutely nothing compared to the massive worldwide effort that many people have to put in to fix the problem of a company’s shitty product and release practices.

This is inexcusable, point blank. “A couple of hours is a long time” is not a valid excuse when the alternative, as clearly evidenced, is millions of computers and critical systems simultaneously failing hard.

This might have been different if it was a small subset of computers, but this clearly could have been caught in minutes with any sort of sensible testing or canary rollout practices.

Re: Technical Details on Today's Outage

#98

Post by mynameisvlad »

gjm11 wrote: post #87
hatsunearu wrote: post #12

Earlier quoted context omitted.

> It doesn't read to me as trying to dodge anything. It absolutely reads like this. They are getting blasted online for shipping kernel mode driver updates without proper QA and release engineering. Which just from face value just seems like some insano style engineering. They are saying "it's not actually a kernel mode value" to deflect blame. I mean, I really don't understand why they would make this statement othe…

They're getting blasted for causing a massive worldwide outage due to what is clearly inadequate quality control. I don't see why this is any better if it's "pushed a kernel-mode driver update with bugs in it" than if it's "released a product with buggy kernel-mode stuff that can be made to crash by an innocuous-looking data file, and then pushed a data file that made it crash". Same result either way. Same demonstra…

You may understand it that way, but you also have a much deeper knowledge of this than the targeted audience of the RCA.

Make no mistake, this RCA was not published for technical folks. The only reason it’s even published is to make their customers feel more secure. You and I are not their customers; high level management and executives are.

Re: Technical Details on Today's Outage

#99

Post by tootie »

hi-v-rocknroll wrote: post #71
PedroBatista wrote: post #34

Light on technical and light on details. Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.

Yup. - They don't do enough or the right kind of smoke tests. - They don't do exponential-canary deployments with an ability to rollback, and instead just YOLO it. - They don't appear to have a customer-side security / client platform team update approval gating change control process for software updates or for definitions (or whatever they use). This is fundamentally laziness and/or incompetency.

They say they run this process multiple times a day. Must be tens of thousands of deployments. I'd guess complacency set in at some point. Just completely inured to the risks they were taking.

Re: Technical Details on Today's Outage

#100

Post by bostik »

hatsunearu wrote: post #12
gjm11 wrote: post #8

Earlier quoted context omitted.

It doesn't read to me as trying to dodge anything. They aren't saying "they're not kernel drivers, so everything is OK", they're saying "seeing the .sys on the filenames, you might think they're kernel drivers, but as it happens they're something else". (Maybe there's some subtext that I'm missing, but I don't see how saying "these aren't kernel drivers" makes them look any better, and I do see why they might say it…

> It doesn't read to me as trying to dodge anything. It absolutely reads like this. They are getting blasted online for shipping kernel mode driver updates without proper QA and release engineering. Which just from face value just seems like some insano style engineering. They are saying "it's not actually a kernel mode value" to deflect blame. I mean, I really don't understand why they would make this statement othe…

Friend does incident response and Windows forensics, and pointed something (in retrospect) rather obvious out yesterday: the instructions for cleaning up simply told people to "delete .SYS files according to this wildcard". No additional context.

That caught his eye, because to him it sounded like madness. Apparently deleting random driver files is a fairly well known way to screw a Windows system up even more than it already was.

This statement from CS must have gone through legal and PR review, so we have to assume every word and statement has been carefully vetted from a cover-your-backside perspective. It is light on information content, but there must be reason for them to so forcefully telegraph that the files being deployed (and removed) are not themselves drivers.

Post reply on HN