Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on Today's Outage

crowdstrike.com

Post reply on HN
Print view View original

121–130 of 136 posts

Re: Technical Details on Today's Outage

#121

Post by kchr »

acdha wrote: post #115
mynameisvlad wrote: post #110

Earlier quoted context omitted.

You’re responding to a hypothetical, not what happened. Let’s say Linux is the leading OS around the world. How can we be sure that they would actually use eBPF if this was the case? They would likely choose the fastest option in order to support the platform as quickly as possible. Perhaps eBPF didn’t even exist if they prioritized Linux support and implemented that first, since Falcon was first released in 2013 and…

They are using eBPF right now. That suggests that they, like everyone else, see benefits in using a platform feature when it exists.

Last time I checked, CS primarily runs in kernel mode on Linux and only fall back to eBPF if the kernel version is not supported. When in eBPF mode, they call it "Reduced Functionality Mode (RFM)".

Has this changed?

Re: Technical Details on Today's Outage

#122

Post by lores »

notepad0x90 wrote: post #118
TheOtherHobbes wrote: post #95

Earlier quoted context omitted.

In which case "Are you awake and sane?" would be a sensible reality check before heading out. You're trying to hand-wave away the inexcusable. The outage is a symptom. The problem is the lack of even the most basic testing. Clearly these files are sent out without even a minimal sanity check. That is a problem, and it's not something that can be hand-waved away.

In the 3-4 decades of the security industry, testing signature files to see if they trigger a corner case system crash has never been practiced. You and others are proclaiming yourselves to be experts in an area of technology you have no experience in. This was not a software update!!

Then that's 3-4 decades of massive incompetence, isn't it? "Testing before pushing an update" is basic engineering, they have a huge scale so huge responsibility, and they have the money to perform the tests and hire people who aren't entirely stupid. That's gross malpractice.

Re: Technical Details on Today's Outage

#123

Post by bostik »

dizhn wrote: post #120
bostik wrote: post #100

Earlier quoted context omitted.

Friend does incident response and Windows forensics, and pointed something (in retrospect) rather obvious out yesterday: the instructions for cleaning up simply told people to "delete .SYS files according to this wildcard". No additional context. That caught his eye, because to him it sounded like madness. Apparently deleting random driver files is a fairly well known way to screw a Windows system up even more than i…

They said to delete a single specific file. Did what you're saying happen before that or something?

The original instructions were "delete [something]00291-*.sys"

Re: Technical Details on Today's Outage

#124

Post by notepad0x90 »

lores wrote: post #122
notepad0x90 wrote: post #118

Earlier quoted context omitted.

In the 3-4 decades of the security industry, testing signature files to see if they trigger a corner case system crash has never been practiced. You and others are proclaiming yourselves to be experts in an area of technology you have no experience in. This was not a software update!!

Then that's 3-4 decades of massive incompetence, isn't it? "Testing before pushing an update" is basic engineering, they have a huge scale so huge responsibility, and they have the money to perform the tests and hire people who aren't entirely stupid. That's gross malpractice.

testing for software, not for content. you test, and fuzz the software that processes the updates, not the content files themselves. it's like a post on HN crashing HN and you claiming HN should have tested each post before allowing it to be displayed. you test code not data, and I dare you to back up any claim that data processed by software should also be tested in the same way. Everyone is suddenly an expert in AV content updates lol.

Re: Technical Details on Today's Outage

#125

Post by ciuokan »

notepad0x90 wrote: post #124
lores wrote: post #122

Earlier quoted context omitted.

Then that's 3-4 decades of massive incompetence, isn't it? "Testing before pushing an update" is basic engineering, they have a huge scale so huge responsibility, and they have the money to perform the tests and hire people who aren't entirely stupid. That's gross malpractice.

testing for software, not for content. you test, and fuzz the software that processes the updates, not the content files themselves. it's like a post on HN crashing HN and you claiming HN should have tested each post before allowing it to be displayed. you test code not data, and I dare you to back up any claim that data processed by software should also be tested in the same way. Everyone is suddenly an expert in AV…

I used to work for Microsoft in a team adjacent to the Defender team that worked on signature updates and I know for sure that these were tested before being rolled out - I saw the Azure Devops pipelines they used to do this. If other companies aren't doing this then that's their incompetence but be assured that it's not industry-wide.

Re: Technical Details on Today's Outage

#126

Post by 0nate »

The lower you go in system architecture, the greater the impact when defects occur. In this instance, the Crowdstrike agent is embedded within the Windows Kernel, and registered with the Kernel Filter Engine illustrated in the diagram below.

https://www.nathanhandy.blog/images/blog/OSI%20Model%20in%20...

If the initial root cause analysis is correct, Crowdstrike has pushed out a bug that could have been easily stopped had software engineering best practices been followed: Unit Testing, Code Coverage, Integration Testing, Definition of Done.

Re: Technical Details on Today's Outage

#127

Post by lores »

notepad0x90 wrote: post #124
lores wrote: post #122

Earlier quoted context omitted.

Then that's 3-4 decades of massive incompetence, isn't it? "Testing before pushing an update" is basic engineering, they have a huge scale so huge responsibility, and they have the money to perform the tests and hire people who aren't entirely stupid. That's gross malpractice.

testing for software, not for content. you test, and fuzz the software that processes the updates, not the content files themselves. it's like a post on HN crashing HN and you claiming HN should have tested each post before allowing it to be displayed. you test code not data, and I dare you to back up any claim that data processed by software should also be tested in the same way. Everyone is suddenly an expert in AV…

A config file IS code. And yes, even a post can theoretically break a site (SQL injection, say), so if you're pushing data to a million PCs you'd better be testing it.

Re: Technical Details on Today's Outage

#128

Post by notepad0x90 »

ciuokan wrote: post #125
notepad0x90 wrote: post #124

Earlier quoted context omitted.

testing for software, not for content. you test, and fuzz the software that processes the updates, not the content files themselves. it's like a post on HN crashing HN and you claiming HN should have tested each post before allowing it to be displayed. you test code not data, and I dare you to back up any claim that data processed by software should also be tested in the same way. Everyone is suddenly an expert in AV…

I used to work for Microsoft in a team adjacent to the Defender team that worked on signature updates and I know for sure that these were tested before being rolled out - I saw the Azure Devops pipelines they used to do this. If other companies aren't doing this then that's their incompetence but be assured that it's not industry-wide.

I'm not saying they don't test them, I'm saying they don't do code tests, as in unit tests and all that. I have no idea what they do, I'm just speculating here, but if in fact they do no testing at all, then I agree that would be pretty bad.I would think their testing would be for how well it detects things and/or performance impact and I'd expect it to be automated deployment (i.e.: test cases are passing = gets deployed), i guess they don't have "did the system crash" check in their pipelines? In your experience at MS, did they test for system/sensor availability impact?

Re: Technical Details on Today's Outage

#129

Post by notepad0x90 »

lores wrote: post #127
notepad0x90 wrote: post #124

Earlier quoted context omitted.

testing for software, not for content. you test, and fuzz the software that processes the updates, not the content files themselves. it's like a post on HN crashing HN and you claiming HN should have tested each post before allowing it to be displayed. you test code not data, and I dare you to back up any claim that data processed by software should also be tested in the same way. Everyone is suddenly an expert in AV…

A config file IS code. And yes, even a post can theoretically break a site (SQL injection, say), so if you're pushing data to a million PCs you'd better be testing it.

You're right, but "testing" could mean anything, you'd need to have the foresight to anticipate the config crashing the program. Is it common to test for that scenario with config files?

Re: Technical Details on Today's Outage

#130

Post by notepad0x90 »

mynameisvlad wrote: post #97
dalyons wrote: post #91

Earlier quoted context omitted.

A couple of hours is a long time in the world of automated attacks

A couple of hours is absolutely nothing compared to the massive worldwide effort that many people have to put in to fix the problem of a company’s shitty product and release practices. This is inexcusable, point blank. “A couple of hours is a long time” is not a valid excuse when the alternative, as clearly evidenced, is millions of computers and critical systems simultaneously failing hard. This might have been diff…

I'm guessing they didn't expect content updates to cause such an impact, they've been doing this for 15 years, it is that uncommon. a couple of hours in their world is a long time because their concern is protecting customers as soon as possible. I'm sure they'll do all kinds of tests going forward and be transparent about it. Keep in mind how easy it is for you or I to come to conclusions without understanding or knowing the context they operate in, maybe it will be more clear soon enough.
Post reply on HN