Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

301–310 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#301

Post by willy_k »

digitalsushi wrote: post #112
taspeotis wrote: post #52

Earlier quoted context omitted.

Personal: Nothing - Windows Defender is built into Windows. Business: Nothing - Windows Defender Advanced Threat Protection is built into the higher Microsoft 365 license tiers. It amazes me people chose to pay money to have all their PCs bluescreen.

if you had used 'some' before 'people' i could agree but some industries have to use a siem or they can be fined, so, i mean if there's a list of siems that are definitely not going to ever crash by messing around in the kernel lets get a list going

Luckily the concern isn’t simply whether they could make a mistake and cause a crash by easing around in the kernel, it’s whether they’re likely to, and I’d argue that CrowdStrike is particularly likely to do so given their testing and rollout processes, and the culture that encompasses those failures

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#302

Post by hello_moto »

sigseg1v wrote: post #226
hello_moto wrote: post #153

Earlier quoted context omitted.

the person you're replying will not take any sane argument once they decided that UX must be involved in kernel technical decision...

How would it not be related? Jamming untested code down the pipe with no way for users to configure when it's deployed and then rendering their machines inoperable is an extremely bad user experience and I would absolutely expect a UX expert to step in to try to avoid that.

Pick any large company that has a division working on Linux kernel (say Android).

I bet my ass UX is not anywhere close to the low-level OS team.

UX is definitely embedded in the App level team but not in low-level.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#303

Post by hello_moto »

tsimionescu wrote: post #264
mewpmewp2 wrote: post #203

Earlier quoted context omitted.

So it would be preferable to have your data encrypted, taken hostage unless you pay, and be down for days, instead of 6 hours of just down?

Do you seriously believe that all CrowdStrike on Windows customers were at such imminent risk of ransomware that one-two hours to run this on one internal setup and catch the critical error they released would have been dangerous? This is a ludicrous position, and has been proven obviously false by the proceedings: all systems that were crashed by this critical failure were not, in fact, attacked with ransomware once…

I'd challenge you to be a CISO :)

You don't want to be in a situation where you're taken hostage and asked hundred mills ransomeware just because you're too slow to mitigate the situation.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#304

Post by roblabla »

busterarm wrote: post #233
roblabla wrote: post #212

Earlier quoted context omitted.

How do you mitigate the OS breaking an API below you in an update? Test the updates before they come out? Even if you could, you'd still need to deploy a fix before the OS update hits the customers, and anyone that didn't update would still be affected. The linux case is just _very_ different from the windows case. The mitigation steps that could have been taken to avoid the linux problem would not have helped for th…

You're missing the forest for the trees. It's: a) an update, b) pushed out globally without proper testing, c) that bricked the OS. It's an obvious failure mode that if you have a proper incident response process would be revealed from that specific incident and flagged for needing mitigation. I do this specific thing for a living. You don't just address the exact failure that happened but try to identify classes of…

Sure, customers are free to blame their vendor. But please, we’re on HN, we aren’t customers, we don’t have beef in this game. So we can do better here, and properly allocate blame, instead of piling on the cs hate for internet clout.

And again, you cannot prevent your vendor breaking you. Sure, you can magic some convoluted process to catch it asap. But that won’t help the poor sods who got caught in-between.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#305

Post by dang »

hulitu wrote: post #295
mewpmewp2 wrote: post #5

Earlier quoted context omitted.

There definitely was a huge outage, but based on the given information we still can't know for sure how much they invested in testing and quality control. There's always a chance of failure even for the most meticulous companies. Now I'm not defending or excusing the company, but a singular event like this can happen to anyone and nothing is 100%. If thorough investigation revealed poor quality control investment com…

[flagged]

Could you please stop posting unsubstantive comments and/or flamebait? Posts like this one and https://news.ycombinator.com/item?id=41542151 are definitely not what we're trying for on HN.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#306

Post by canucker2016 »

hello_moto wrote: post #151
canucker2016 wrote: post #85

Earlier quoted context omitted.

They literally half-assed their deployment process - one part enterprisey, one part "move fast and break things". Guess which part took down much of the corporate world? from Preliminary Post Incident Review at https://www.crowdstrike.com/falcon-content-update-remediatio... : "CrowdStrike delivers security content configuration updates to our sensors in two ways: Sensor Content that is shipped with our sensor directl…

> one part enterprisey, one part "move fast and break things". When there's 0day, how enterprisey you would like to catch the 0day?

Crowdstrike exploited their own 0-day. Their market cap went down by several billion dollars.

A patch should, at minimum:

1. Let the app run 2a. Block the offending behaviour 2b. Allow normal behaviour

Part 1. can be assumed if Parts 2a and 2b work correctly.

We know CrowdStrike didn't ensure 2a or 2b since the app caused the machine to reboot when the patch caused a fault in the app.

CrowdStrike's Root Cause Analysis, https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann..., lists what they're going to do:

====

Mitigation: Validate the number of input fields in the Template Type at sensor compile time

Mitigation: Add runtime input array bounds checks to the Content Interpreter for Rapid Response Content in Channel File 291 - An additional check that the size of the input array matches the number of inputs expected by the Rapid Response Content was added at the same time. - We have completed fuzz testing of the Channel 291 Template Type and are expanding it to additional Rapid Response Content handlers in the sensor.

Mitigation: Correct the number of inputs provided by the IPC Template Type

Mitigation: Increase test coverage during Template Type development

Mitigation: Create additional checks in the Content Validator

Mitigation: Prevent the creation of problematic Channel 291 files

Mitigation: Update Content Configuration System test procedures

Mitigation: The Content Configuration System has been updated with additional deployment layers and acceptance checks

Mitigation: Provide customer control over the deployment of Rapid Response Content updates

====

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#307

Post by Fnoord »

notepad0x90 wrote: post #282
Fnoord wrote: post #208

Earlier quoted context omitted.

Sure, storage. Networking though? SIEMs receive and send data unencrypted? They should not. By sending the data in plain text you open up an attack surface to anyone sniffing the network.

Crowdstrike like many EDRs uses mutually authenticated TLS to send the data over the network to their cloud.

OP said:

> Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text.

So this person claims the data isn't send using TLS. I didn't verify the claim myself though.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#308

Post by stogot »

cj wrote: post #131
stogot wrote: post #118

Earlier quoted context omitted.

Exporting to a SIEM does not correlate to either of those extremes. It’s stupidity and makes auditing worse

SIEM = Security Information & Event Management Factually, it is necessary for auditing and absolutely correlates with the extreme of needing to monitor the “usage” of “secrets”. In a highly auditable/“secure” environment, you can’t give secrets to employees with no tracking of when the secrets are used.

I’m well aware of what a SIEM does. You do not need to log a plaintext secret to know what the principal is doing with it. In a highly auditable environment (your words) this is a disaster

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#309

Post by Thompson88 »

Clearly there weren't any code review workflow processes in place, which is astonishing. That's why our primary focus is-transparency, accountability, and system integrity to bring a decentralized, transparent, and reliable platform for journalists, researchers, scientists, and content creators.
Post reply on HN