Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

271–280 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#271

Post by acdha »

esperent wrote: post #220
acdha wrote: post #89

Earlier quoted context omitted.

They probably weren’t, but that still speaks to their general culture and is compatible with what we know about their kernel engineering culture (limited testing, no review, no use of common fail safe mechanisms).

> is compatible with what we know In other words, it confirms our biases and we're willing to accept it at face value despite there being only a single anecdotal piece of evidence.

It sounds like you might want to read their technical report. That’s neither anecdotal nor a single point, and it showed a pretty large gap in engineering leadership with numerous areas well behind the state of the art.

That’s why I said it was compatible: both these former employees and their own report showed an emphasis on shipping rapidly but not the willingness to invest serious money in the safeguards needed to do so safely. If you want to construct another theory, feel free to do so.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#272

Post by GuB-42 »

Izkata wrote: post #269
GuB-42 wrote: post #222

Earlier quoted context omitted.

It could have been ok to expedite data updates, should the code treat configuration data as untrusted input, as if it could be written by an attacker. It means fuzz testing and all that. Obviously the system wasn't very robust, as a simple, within specs change could break it. A company like CrowdStrike, which routinely deals with memory exploits and claims to do "zero trust" should know better. As often, there is a g…

> Obviously the system wasn't very robust, as a simple, within specs change could break it. From my limited understanding, the file was corrupted in some way. Lots of NULL bytes, something like that.

From the report, it seems the problem is that they added a feature that could use 21 arguments, but there was only enough space for 20. Until now, no configuration used all 21 (the last one was a wildcard regex, which apparently didn't count), but when they finally did, it caused a buffer overflow and crashed.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#273

Post by acdha »

Izkata wrote: post #269
GuB-42 wrote: post #222

Earlier quoted context omitted.

It could have been ok to expedite data updates, should the code treat configuration data as untrusted input, as if it could be written by an attacker. It means fuzz testing and all that. Obviously the system wasn't very robust, as a simple, within specs change could break it. A company like CrowdStrike, which routinely deals with memory exploits and claims to do "zero trust" should know better. As often, there is a g…

> Obviously the system wasn't very robust, as a simple, within specs change could break it. From my limited understanding, the file was corrupted in some way. Lots of NULL bytes, something like that.

That rumor floated around Twitter but the company quickly disavowed it. The problem was that they added an extra parameter to a common function but never tested it with a non-wildcard value, revealing a gap in their code coverage review:

https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann...

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#274

Post by righthand »

hitekker wrote: post #239

I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of o…

Former QA engineer here, and can confirm quality is seen as an annoying roadblock in the way of self-interested workers, disguised as in the way of having fun on the customers dime.

My favorite repeated reorg strategy over the years is “that we will train everyone in engineering to be hot swappable in their domains”. Talk about spinning wheels.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#275

Post by ClickedUp »

hitekker wrote: post #239

I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of o…

This is not a game. I would normally agree but not when it comes to low-level kernel drivers. They're a cyber security company making it even worse.

Not very long ago we had this client who ordered a custom high security solution (using a kernel driver). I can't reveal too much but basically they had this offline computer running this critical database and they needed a way to account for every single system call to guarantee that any data could have not been changed without the security system alerting and logging the exact change. No backups etc were allowed to leave the computer ever. We were even required to check ntdll (this was on Windows) for hooks before installing the driver on-site & other safety precautions. Exceptions, freezes or a deadlock? No way. Any system call missed = disaster.

We took this seriously. Whenever we made a change to the driver code we had to re-test the driver on 7 different computers (in-office) running completely different hardware doing a set test procedure. Last test before release entailed an even more extensive test procedure.

This may sound harsh but CrowdStrike are total amateurs, always been. Besides, what have they contributed to the cyber security community? - Nothing! Their research are at a level of a junior cyber security researcher. They are willing to outright lie and jump to wild conclusions which is very frowned upon in the community. Also heard others comment on how CS really doesn't really fit the mold of a standard cyber security company.

Nah, CS should take a close look at true professional companies like Kaspersky and Checkpoint; industry leaders who've created proven top notch security solutions (software/services) but not least actually contributed their valuable research to the community for free, catching zero-days, reporting them before no one even had a chance of exploiting them.

They deserve some criticism.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#276

Post by acdha »

mewpmewp2 wrote: post #31
dartos wrote: post #23

Earlier quoted context omitted.

Totally agree. I’d consider staggering a rollout to be the absolute basics of due diligence. Especially when you’re building a critical part of millions of customer machines.

I would say that canary release is an absolute must 100%. Except I can think of cases where it might still not be enough. So, I just don't feel comfortable judging them out of the box. Does all the evidence seem to point against them? For sure. But I just don't feel comfortable giving that final verdict without knowing for sure. Specifically because this is about fighting against malicious actors, where time can be o…

> Specifically because this is about fighting against malicious actors, where time can be of essence to deploy some sort of protection against a novel threat.

This is severely overstating the problem: an extra few minutes is not going to be the difference between their customers being compromised. Most of the devices they run on are never compromised, because anyone remotely serious has defense in depth.

If it was true, or even close to true, that would make the criticism more rather than less strong. If time is of the essence, you invest in things like reviewing test coverage (their most glaring lapse), fuzz testing, and common reliability engineering techniques like having the system roll back to the last known good configuration after it’s failed to load. We think of progressive rollouts as common now but they got to get that mainstream in large part because the Google Chrome team realized rapid updates are important but then asked what they needed to do to make them safe. CrowdStrike’s report suggests that they wanted rapid but weren’t willing to invest in the implementation because that isn’t a customer-visible feature – until it very painfully became one.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#277

Post by bitcharmer »

pclmulqdq wrote: post #263
bitcharmer wrote: post #238

Earlier quoted context omitted.

We have circuit breakers for that very purpose. Everyone on the street does. It's just that theirs seems to have failed for some reason.

Theirs didn't fail, and they did have one. The circuit breaker they had that would have worked was a big red button that killed all of their trading processes, which would have meant spending the rest of the day figuring out and unwinding their positions. Ihey were unwilling to push that button in the short time they had. If you read the reports to the SEC or the articles about it, you will note that. The follow-ups…

Gotcha, thanks for correcting me, I need to read up more about the incident.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#278

Post by musicale »

ClickedUp wrote: post #275
hitekker wrote: post #239

I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of o…

This is not a game. I would normally agree but not when it comes to low-level kernel drivers. They're a cyber security company making it even worse. Not very long ago we had this client who ordered a custom high security solution (using a kernel driver). I can't reveal too much but basically they had this offline computer running this critical database and they needed a way to account for every single system call to…

I'm don't Kaspersky and Checkpoint either. But CS should exit the market.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#279

Post by fzeroracer »

newshackr wrote: post #120
sonofhans wrote: post #104

Earlier quoted context omitted.

A UX designer might have told them it was a bad idea to deploy the patch widely without testing a smaller cohort, for instance. That’s an obvious measure that they skipped this time.

But that doesn't have anything to do with what UX designers typically do

I can't believe people on HN are posting this stuff over and over again. Either you are holistically disconnected from what proper software development should look like or outright creating the same environments that resulted in the crowdstrike issue.

Software security and quality is the responsibility of everyone on the team. A good UX designer should be thinking of ways a user can escape the typical flow or operate in unintended ways and express that to testers. And in decisions where management is forcing untested patches everyone should chime in.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#280

Post by paulcole »

insane_dreamer wrote: post #256
paulcole wrote: post #98

Earlier quoted context omitted.

Why would you trust a company no-man any more than a company yes-man? They both have agendas and biases. Is it just that you personally prefer one set of biases (anti-company) more than the other (pro-company)?

Well, in this case, we know one side (pro-company) fucked up big time. The other side (anti-company) may or may not have fucked up. That makes it easier to trust one side over another.

You’ve kind of set yourself up in a no-lose situation here.

If the employees fucked up then you’ll say the company still fucked up because it wasn’t managing the employees well.

And then in that situation you’ll still believe the lying employees who say its the company’s fault while leaving out their culpability.

Post reply on HN