Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

81–90 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#81

Post by llm_trw »

hn_throwaway_99 wrote: post #70
daedrdev wrote: post #8

Earlier quoted context omitted.

Two things are clear though Nobody ran this update The update was pushed globally to all computers With that alone we know they have failed the simplest of quality control methods for a piece of software as widespread as theirs. This is even excluding that there should have been some kind of error handling to allow the computer to boot if they did push bad code.

While I agree with this, from a software engineering perspective I think it's more useful to look at the lessons learned. I think it's too easy to just throw "Crowdstrike is a bunch of idiots" against the wall, and I don't think that's true. It's clear to me that CrowdStrike saw this as a data update vs. a code update, and that they had much more stringent QA procedures for code updates that they did data updates. It…

I'm sorry but there comes a point where you have to call a spade a spade.

When you have the trifecta of regex, *argv packing and uninitialized memory you're reaching levels of incompetence which require being actively malicious and not just stupid.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#82

Post by AmericanChopper »

apimade wrote: post #43
st3fan wrote: post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

Is this really a criticism? Because this has been the case forever with all security and SIEM tools. It’s one of the reasons why the SIEM is the most locked down pieces of software in the business. Realistically, secrets alone shouldn’t allow an attacker access - they should need access to infrastructure or a certificates in machines as well. But unfortunately that’s not the case for many SaaS vendors.

Keeping secrets and other sensitive data out of your SIEM is a very important part of SIEM design. Depending on what you’re dealing with you might want to tokenize it, or redact it, but you absolutely don’t want to don’t want to just ingest them in plaintext.

If you’re a PCI company then ending up with a credit card number in your SIEM can be a massive disaster. Because you’re never allowed to store that in plaintext, and your SIEM data is supposed to be immutable. In theory that puts you out of compliance for a minimum of one year with no way to fix it, in reality your QSAs will spend some time debating what to do about it and then require you to figure out some way to delete it, which might be incredibly onerous. But I have no idea what they’d do if your SIEM somehow became full of credit card numbers, that probably is unfixable…

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#83

Post by owl57 »

0xbadcafebee wrote: post #16

Critical software infrastructure should be regulated the way critical physical infrastructure is. We don't trust the people who make buildings and bridges to "do the right thing" - we mandate it with regulations and inspections. (When your software not working strands millions of people around the globe, it's critical) And this was just a regular old "accident"; imagine the future, when a war has threat actors trying…

Did you notice that the piece of software in question was apparently installed mostly in companies where regulations and inspections already override sysadmins' common sense? Are you sure the answer is simply more of the same?

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#84

Post by cj »

ants_everywhere wrote: post #75
apimade wrote: post #43

Earlier quoted context omitted.

Is this really a criticism? Because this has been the case forever with all security and SIEM tools. It’s one of the reasons why the SIEM is the most locked down pieces of software in the business. Realistically, secrets alone shouldn’t allow an attacker access - they should need access to infrastructure or a certificates in machines as well. But unfortunately that’s not the case for many SaaS vendors.

Ideally secrets never leave secure enclaves and humans at the organization can't even access them. It's totally insane to send them to a remote service controlled by another organization.

Essentially, it’s straddling two extremes:

1) employees are trusted with secrets, so we have to audit that employees are treating those secrets securely (via tracking, monitoring, etc)

2) we don’t allow employees to have access to secrets whatsoever, therefore we don’t need any auditing or monitoring

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#85

Post by canucker2016 »

idkwhatimdoin wrote: post #18
mewpmewp2 wrote: post #5

Earlier quoted context omitted.

There definitely was a huge outage, but based on the given information we still can't know for sure how much they invested in testing and quality control. There's always a chance of failure even for the most meticulous companies. Now I'm not defending or excusing the company, but a singular event like this can happen to anyone and nothing is 100%. If thorough investigation revealed poor quality control investment com…

> If thorough investigation revealed poor quality control investment compared to what would be appropriate for a company like this, then we can say for sure. We don't really need that thorough of an investigation. They had no staged deploys when servicing millions of machines. That alone is enough to say they're not running the company correctly.

They literally half-assed their deployment process - one part enterprisey, one part "move fast and break things".

Guess which part took down much of the corporate world?

from Preliminary Post Incident Review at https://www.crowdstrike.com/falcon-content-update-remediatio... :

"CrowdStrike delivers security content configuration updates to our sensors in two ways: Sensor Content that is shipped with our sensor directly, and Rapid Response Content that is designed to respond to the changing threat landscape at operational speed.

...

The sensor release process begins with automated testing, both prior to and after merging into our code base. This includes unit testing, integration testing, performance testing and stress testing. This culminates in a staged sensor rollout process that starts with dogfooding internally at CrowdStrike, followed by early adopters. It is then made generally available to customers. Customers then have the option of selecting which parts of their fleet should install the latest sensor release (‘N’), or one version older (‘N-1’) or two versions older (‘N-2’) through Sensor Update Policies.

The event of Friday, July 19, 2024 was not triggered by Sensor Content, which is only delivered with the release of an updated Falcon sensor. Customers have complete control over the deployment of the sensor — which includes Sensor Content and Template Types.

...

Rapid Response Content is used to perform a variety of behavioral pattern-matching operations on the sensor using a highly optimized engine.

Newly released Template Types are stress tested across many aspects, such as resource utilization, system performance impact and event volume. For each Template Type, a specific Template Instance is used to stress test the Template Type by matching against any possible value of the associated data fields to identify adverse system interactions.

Template Instances are created and configured through the use of the Content Configuration System, which includes the Content Validator that performs validation checks on the content before it is published.

On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data.

Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production."

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#86

Post by pclmulqdq »

0cf8612b2e1e wrote: post #56
ForOldHack wrote: post #45

Earlier quoted context omitted.

Knight Capitol: "$10 million a minute. That’s about how much the trading problem that set off turmoil on the stock market on Wednesday morning is already costing the trading firm. The Knight Capital Group announced on Thursday that it lost $440 million when it sold all the stocks it accidentally bought Wednesday morning because a computer glitch. " Glitch. Oh... https://en.wikipedia.org/wiki/Therac-25

I do not work in finance, but surely every trading company has had an algorithm go wild at some point. Just becomes a matter of how fast someone can pull the circuit breaker before the expensive failure becomes public.

Shamelessly plugging my own blog post on this: https://specbranch.com/posts/knight-capital/

The TL;DR of Knight is that Knight had several things go wrong at the same time, and had no circuit breaker for the problem that did not stop trading for the whole firm for the day. Most trading firms have had things go badly, but the holes in the Swiss cheese aligned for Knight (and they were larger than many other firms). This all comes from a sort of culture of carelessness.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#87

Post by st3fan »

llm_trw wrote: post #79
gruez wrote: post #71

Earlier quoted context omitted.

Otherwise malware can hide in environment variables

Ok, suppose you're right. Why are they only doing it for macs then?

It may depend a bit on your organization but I bet most folks using an EDR solution can tell you that Macs are probably very low on the list when it comes to malware. You can guess which OS you will spend time on every day ...

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#88

Post by beachy »

hn_throwaway_99 wrote: post #70
daedrdev wrote: post #8

Earlier quoted context omitted.

Two things are clear though Nobody ran this update The update was pushed globally to all computers With that alone we know they have failed the simplest of quality control methods for a piece of software as widespread as theirs. This is even excluding that there should have been some kind of error handling to allow the computer to boot if they did push bad code.

While I agree with this, from a software engineering perspective I think it's more useful to look at the lessons learned. I think it's too easy to just throw "Crowdstrike is a bunch of idiots" against the wall, and I don't think that's true. It's clear to me that CrowdStrike saw this as a data update vs. a code update, and that they had much more stringent QA procedures for code updates that they did data updates. It…

> It's clear to me that CrowdStrike saw this as a data update vs. a code update, and that they had much more stringent QA procedures for code updates that they did data updates.

It cannot have been a surprise to Crowdstrike that pushing bad data had the potential to bork the target computer. So if they had such an attitude that would indicate striking incompetence. So perhaps you are right.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#89

Post by acdha »

avree wrote: post #41

"“Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” Their 'expert' on engineering process is a senior UX designer? Somehow, I doubt they were very close to the kernel patch deployment process.

They probably weren’t, but that still speaks to their general culture and is compatible with what we know about their kernel engineering culture (limited testing, no review, no use of common fail safe mechanisms).

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#90

Post by Aeolun »

apimade wrote: post #43
st3fan wrote: post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

Is this really a criticism? Because this has been the case forever with all security and SIEM tools. It’s one of the reasons why the SIEM is the most locked down pieces of software in the business. Realistically, secrets alone shouldn’t allow an attacker access - they should need access to infrastructure or a certificates in machines as well. But unfortunately that’s not the case for many SaaS vendors.

If my security software exfiltrates my secrets by design, I’m just going to give up on keeping anything secure now.
Post reply on HN