Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

151–160 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#151

Post by hello_moto »

canucker2016 wrote: post #85
idkwhatimdoin wrote: post #18

Earlier quoted context omitted.

> If thorough investigation revealed poor quality control investment compared to what would be appropriate for a company like this, then we can say for sure. We don't really need that thorough of an investigation. They had no staged deploys when servicing millions of machines. That alone is enough to say they're not running the company correctly.

They literally half-assed their deployment process - one part enterprisey, one part "move fast and break things". Guess which part took down much of the corporate world? from Preliminary Post Incident Review at https://www.crowdstrike.com/falcon-content-update-remediatio... : "CrowdStrike delivers security content configuration updates to our sensors in two ways: Sensor Content that is shipped with our sensor directl…

> one part enterprisey, one part "move fast and break things".

When there's 0day, how enterprisey you would like to catch the 0day?

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#152

Post by userbinator »

cj wrote: post #84
ants_everywhere wrote: post #75

Earlier quoted context omitted.

Ideally secrets never leave secure enclaves and humans at the organization can't even access them. It's totally insane to send them to a remote service controlled by another organization.

Essentially, it’s straddling two extremes: 1) employees are trusted with secrets, so we have to audit that employees are treating those secrets securely (via tracking, monitoring, etc) 2) we don’t allow employees to have access to secrets whatsoever, therefore we don’t need any auditing or monitoring

employees are trusted with secrets, so we have to audit that employees are treating those secrets securely

IMHO needing to be monitored constantly is not being "trusted" by any sense of the word.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#153

Post by hello_moto »

newshackr wrote: post #120
sonofhans wrote: post #104

Earlier quoted context omitted.

A UX designer might have told them it was a bad idea to deploy the patch widely without testing a smaller cohort, for instance. That’s an obvious measure that they skipped this time.

But that doesn't have anything to do with what UX designers typically do

the person you're replying will not take any sane argument once they decided that UX must be involved in kernel technical decision...

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#154

Post by hello_moto »

acdha wrote: post #89
avree wrote: post #41

"“Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” Their 'expert' on engineering process is a senior UX designer? Somehow, I doubt they were very close to the kernel patch deployment process.

They probably weren’t, but that still speaks to their general culture and is compatible with what we know about their kernel engineering culture (limited testing, no review, no use of common fail safe mechanisms).

A company can have different business units with different culture/mentality.

I bet my ass anyone working in low-level code don't ship the way you do in Cloud.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#155

Post by hello_moto »

hyperpape wrote: post #65
panic wrote: post #48

Why would it matter? The absolute worst case scenario happened and their stock is still up 50% YoY, beating the S&P 500.

It’s down 30% since the incident, and flat since 3 years ago. If it runs up a huge amount in the first half of the year and then the incident knocks off 30% of their market, that still means the incident was really bad.

Their stock has always been volatile but you can't ignore the fact that it hasn't been that bad after the incident.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#156

Post by notepad0x90 »

st3fan wrote: post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

that's what EDRs do. anyone with access to your SIEM or CS data should also be trusted with response access (i.e.: remotely access those machines).

If you want this redacted, it is a SIEM functionality not Crowdstrike's. Depends on the SIEM but even older generation SIEMs have a data scrubbing feature.

This isn't a Crowdstrike design decision as you've put it. any endpoint monitoring too, including the free and open source ones behave just as you described. You won't just see env vars from macs but things like domain admin creds and PKI root signing private keys. If you give someone access to an EDR, or they are incident responders with SIEM access, you've trusted them with full -- yet, auditable and monitored -- access to that deployment.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#157

Post by notepad0x90 »

hiddencost wrote: post #150
st3fan wrote: post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

This kind of information seems like it should have a CVE and a responsible disclosure process. Kidding, mostly, but wow that's a hell of a vulnerability.

It is not a vulnerability, you literally pay for this feature. I really don't want to defend Crowdstrike but HN keeps making it hard not to.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#158

Post by fragmede »

userbinator wrote: post #152
cj wrote: post #84

Earlier quoted context omitted.

Essentially, it’s straddling two extremes: 1) employees are trusted with secrets, so we have to audit that employees are treating those secrets securely (via tracking, monitoring, etc) 2) we don’t allow employees to have access to secrets whatsoever, therefore we don’t need any auditing or monitoring

employees are trusted with secrets, so we have to audit that employees are treating those secrets securely IMHO needing to be monitored constantly is not being "trusted" by any sense of the word.

I can trust you enough to let you borrow my car and not crash it, but still want to know where my car is with an Airtag.

Similarly employees can be trusted enough with access to prod, while the company wants to protect itself from someone getting phished or from running the wrong "curl | bash" command, so the company doesn't get pwned.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#159

Post by notepad0x90 »

skewer99 wrote: post #74
st3fan wrote: post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

The monitoring and collection isn't the problem, that's what modern EDR does - collect, analyze, compare, and do statistics on all of the things. The plaintext part is not okay.

Thank you, that's a sound perspective, but it is the responsibility of the security staff who deploy EDRs like Crowdstrike to scrub any data at ingestion time into their SIEM. but within CS's platform, it makes little sense to talk about scrubbing, since CS doesn't know what you want scrubbed unless it is standardized data forms (like SSNs,credit cards,etc..).

Another way to look at it is, the CS cloud environment is effectively part of your environment. the secrets can get scrubbed, but CS still has access to your devices, they can remotely access them and get those secrets at any time without your knowledge. that is the product. The security boundary of OP's mac is inclusive of the CS cloud.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#160

Post by Comma2976 »

hn_throwaway_99 wrote: post #70
daedrdev wrote: post #8

Earlier quoted context omitted.

Two things are clear though Nobody ran this update The update was pushed globally to all computers With that alone we know they have failed the simplest of quality control methods for a piece of software as widespread as theirs. This is even excluding that there should have been some kind of error handling to allow the computer to boot if they did push bad code.

While I agree with this, from a software engineering perspective I think it's more useful to look at the lessons learned. I think it's too easy to just throw "Crowdstrike is a bunch of idiots" against the wall, and I don't think that's true. It's clear to me that CrowdStrike saw this as a data update vs. a code update, and that they had much more stringent QA procedures for code updates that they did data updates. It…

Crowdstrike is a bunch of idiots
Post reply on HN