Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

221–230 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#221

Post by diatone »

newshackr wrote: post #120
sonofhans wrote: post #104

Earlier quoted context omitted.

A UX designer might have told them it was a bad idea to deploy the patch widely without testing a smaller cohort, for instance. That’s an obvious measure that they skipped this time.

But that doesn't have anything to do with what UX designers typically do

Not true; UX designers typically are responsible for advocating for a robust, intuitive experience for users. The fact that kernel updates don’t have a user interface doesn’t make them exempt from asking the simple question: how will this affect users? And the subsequent question: is there a chance that deploying this eviscerates the user experience?

Granted, a company that isn’t focused on the user experience as much as it is on other things might not prioritise this as much in the first place.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#222

Post by GuB-42 »

hn_throwaway_99 wrote: post #70
daedrdev wrote: post #8

Earlier quoted context omitted.

Two things are clear though Nobody ran this update The update was pushed globally to all computers With that alone we know they have failed the simplest of quality control methods for a piece of software as widespread as theirs. This is even excluding that there should have been some kind of error handling to allow the computer to boot if they did push bad code.

While I agree with this, from a software engineering perspective I think it's more useful to look at the lessons learned. I think it's too easy to just throw "Crowdstrike is a bunch of idiots" against the wall, and I don't think that's true. It's clear to me that CrowdStrike saw this as a data update vs. a code update, and that they had much more stringent QA procedures for code updates that they did data updates. It…

It could have been ok to expedite data updates, should the code treat configuration data as untrusted input, as if it could be written by an attacker. It means fuzz testing and all that.

Obviously the system wasn't very robust, as a simple, within specs change could break it. A company like CrowdStrike, which routinely deals with memory exploits and claims to do "zero trust" should know better.

As often, there is a good chance it is an organization problem. The team in charge of the parsing expected that the team in charge of the data did their tests and made sure the files weren't broken, while on the other side, they expected the parser to be robust and at worst, a quick rollback could fix the problem. This may indeed be the sign of a broken company culture, which would give some credit to the ex-employees.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#225

Post by lr4444lr »

Alupis wrote: post #2

> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card. Look, I think just as poorly about CrowdStr…

In principle yes, I agree that former employees' sentiments have an obvious bias, but if they all trend in the same direction - people who worked in different times and functions and didn't know each other while on the job - that points to a likely underlying truth.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#226

Post by sigseg1v »

hello_moto wrote: post #153
newshackr wrote: post #120

Earlier quoted context omitted.

But that doesn't have anything to do with what UX designers typically do

the person you're replying will not take any sane argument once they decided that UX must be involved in kernel technical decision...

How would it not be related? Jamming untested code down the pipe with no way for users to configure when it's deployed and then rendering their machines inoperable is an extremely bad user experience and I would absolutely expect a UX expert to step in to try to avoid that.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#227

Post by [deleted] »

notepad0x90 wrote: post #156
st3fan wrote: post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

that's what EDRs do. anyone with access to your SIEM or CS data should also be trusted with response access (i.e.: remotely access those machines). If you want this redacted, it is a SIEM functionality not Crowdstrike's. Depends on the SIEM but even older generation SIEMs have a data scrubbing feature. This isn't a Crowdstrike design decision as you've put it. any endpoint monitoring too, including the free and open…

[deleted]

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#228

Post by sersi »

Crowdstrike was heavily pushed on us at a previous company both for compliance reason by some of our clients (BCG were the ones pushing us to use crowdstrike) and from our liability insurance company.

It was really an uphill battle to convince everyone not to use Crowdstrike. Eventually I managed to but after many meetings where I had to spend a significant amount of time convincing different shareholders. I'm sure a lot of people just fold and go with them.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#229

Post by mikeocool »

sersi wrote: post #228

Crowdstrike was heavily pushed on us at a previous company both for compliance reason by some of our clients (BCG were the ones pushing us to use crowdstrike) and from our liability insurance company. It was really an uphill battle to convince everyone not to use Crowdstrike. Eventually I managed to but after many meetings where I had to spend a significant amount of time convincing different shareholders. I'm sure a…

Curious — did you go with a different EDR solution? Or were you able to convince people not to roll one out at all?

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#230

Post by odyssey7 »

pclmulqdq wrote: post #86
0cf8612b2e1e wrote: post #56

Earlier quoted context omitted.

I do not work in finance, but surely every trading company has had an algorithm go wild at some point. Just becomes a matter of how fast someone can pull the circuit breaker before the expensive failure becomes public.

Shamelessly plugging my own blog post on this: https://specbranch.com/posts/knight-capital/ The TL;DR of Knight is that Knight had several things go wrong at the same time, and had no circuit breaker for the problem that did not stop trading for the whole firm for the day. Most trading firms have had things go badly, but the holes in the Swiss cheese aligned for Knight (and they were larger than many other firms). Th…

I always thought the Swiss cheese model was used to suggest that no one party could possibly be responsible for a bad thing that happened. Interesting to see the company’s culture blamed for the cheese itself.
Post reply on HN