Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

41–50 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#41
"“Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.”

Their 'expert' on engineering process is a senior UX designer? Somehow, I doubt they were very close to the kernel patch deployment process.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#42
post #8

Earlier quoted context omitted.

Two things are clear though Nobody ran this update The update was pushed globally to all computers With that alone we know they have failed the simplest of quality control methods for a piece of software as widespread as theirs. This is even excluding that there should have been some kind of error handling to allow the computer to boot if they did push bad code.

Also it's the _second_ time that they had done this in a few short months. They had previous bricked linux hosts earlier with a similar type of update. So we also know that they don't learn from their mistakes.

The blame for the Linux situation isn’t as clear cut as you make it out to be. Red hat rolled out a breaking change to BPF which was likely a regression. That wasn’t caused directly by a crowdstrike update.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#43
post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

Is this really a criticism? Because this has been the case forever with all security and SIEM tools. It’s one of the reasons why the SIEM is the most locked down pieces of software in the business.

Realistically, secrets alone shouldn’t allow an attacker access - they should need access to infrastructure or a certificates in machines as well. But unfortunately that’s not the case for many SaaS vendors.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#44
post #43
post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

Is this really a criticism? Because this has been the case forever with all security and SIEM tools. It’s one of the reasons why the SIEM is the most locked down pieces of software in the business. Realistically, secrets alone shouldn’t allow an attacker access - they should need access to infrastructure or a certificates in machines as well. But unfortunately that’s not the case for many SaaS vendors.

But why only forced on MacOS?

I think some configurability would be great. I would like to provide an allow list or the ability to redact. Or exclude specific host groups.

We all have different levels of acceptable risk

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#45

Everything that we know about CrowdStrike stinks of Knight Capital to me. A minor culture problem snowballed into complete dysfunction, eventually resulting in a company-ending bug.

Knight Capitol:

"$10 million a minute.

That’s about how much the trading problem that set off turmoil on the stock market on Wednesday morning is already costing the trading firm.

The Knight Capital Group announced on Thursday that it lost $440 million when it sold all the stocks it accidentally bought Wednesday morning because a computer glitch. "

Glitch. Oh...

https://en.wikipedia.org/wiki/Therac-25

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#46
post #42

Earlier quoted context omitted.

Also it's the _second_ time that they had done this in a few short months. They had previous bricked linux hosts earlier with a similar type of update. So we also know that they don't learn from their mistakes.

The blame for the Linux situation isn’t as clear cut as you make it out to be. Red hat rolled out a breaking change to BPF which was likely a regression. That wasn’t caused directly by a crowdstrike update.

At least one of the incidents involved Debian machines, so I don't understand how Red Hat's change would be related.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#47
post #42

Earlier quoted context omitted.

The blame for the Linux situation isn’t as clear cut as you make it out to be. Red hat rolled out a breaking change to BPF which was likely a regression. That wasn’t caused directly by a crowdstrike update.

At least one of the incidents involved Debian machines, so I don't understand how Red Hat's change would be related.

Sorry, that’s correct it was Debian, but Debian did apply a RHEL specific patch to their kernel. That’s the relationship to red hat.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#49
post #43
post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

Is this really a criticism? Because this has been the case forever with all security and SIEM tools. It’s one of the reasons why the SIEM is the most locked down pieces of software in the business. Realistically, secrets alone shouldn’t allow an attacker access - they should need access to infrastructure or a certificates in machines as well. But unfortunately that’s not the case for many SaaS vendors.

The certificate private key is also a secret.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#50
post #2

> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card. Look, I think just as poorly about CrowdStr…

This is like online reviews. If you selectively take positive or negative reviews and somehow censor the rest, the reviews are worthless. Yet, if you report on all the ones you find, it's still useful.

Yes, I'm more likely to leave reviews if I'm unsatisfied. Yes, people are more likely to leave CS if they were unhappy. Biased data, but still useful data.

Post reply on HN