Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

281–290 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#281

Earlier quoted context omitted.

I've worked in these enterprise organizations for a long time. They don't run on common sense, or even what one might consider "business sense". Their existing incentives create bizarre behavior. For example, you might think "if a big security exploit happens, the stock price might tank" . So if they value the stock price, they'll focus on security, right?. In reality what they do is focus on burying the evidence of…

Probably there should be an independent body that oversees postmortems on tech issues, with the ability to suggest changes. This is what airlines face during crash investigations and often new rules are put in place (e.g., don’t let the shift manager self-certify his own work in the incident where the pilot’s window popped off). How this would look like with software companies, and what the bar is for being subject t…

I've commented on this before, but in this case I think it starts to fall onto the laps of the individual employees themselves by way of licensing, or at least some sort of certification system. Sure, you could skirt a test here or there, but then you'd only be shorting yourself when shit hits the fan. It'd be your license and essentially your livelihood on the line.

"Proper" engineering disciplines have similar systems like the Professional Engineer cert via the NSPE that requires designs be signed off. If you had the requirement that all software engineers (now with the certification actually bestowing them the proper title of 'engineer') sign off on their design, you could prevent the company from just finding someone else more unscrupulous to push that update or whatever through. If the entirety of the department or company is employing properly certificated people, they'd be stuck actually doing it the right way.

That's their incentive to do it correctly: sign your name to it, or lose your license, and just for drama's sake, don't collect $200, directly to jail. For the companies, employ properly licensed engineers, or risk unlimited downside liability when shit goes sideways, similar to what might happen if an engineering firm built a shoddy bridge.

Would a firm that peddles some sort of CRUD app need to go through all of this? If it handles toxic data like payments or health data or other PII, sure. Otherwise, probably not, just like you have small contracting outfits that build garden sheds or whatever being a bit different than those that maintain, say, cooling systems for nuclear plants. Perhaps a law might be written to include companies that work in certain industries or business lines to compel them to do this.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#282
post #208

Earlier quoted context omitted.

that's what EDRs do. anyone with access to your SIEM or CS data should also be trusted with response access (i.e.: remotely access those machines). If you want this redacted, it is a SIEM functionality not Crowdstrike's. Depends on the SIEM but even older generation SIEMs have a data scrubbing feature. This isn't a Crowdstrike design decision as you've put it. any endpoint monitoring too, including the free and open…

Sure, storage. Networking though? SIEMs receive and send data unencrypted? They should not. By sending the data in plain text you open up an attack surface to anyone sniffing the network.

Crowdstrike like many EDRs uses mutually authenticated TLS to send the data over the network to their cloud.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#283
Just another example of technical leadership being completely irresponsible and another example of tech companies prioritizing the wrong things. As a security company, this completely blows their credibility. i’m not convinced they learned anything from this and don’t expect this effect to change anything. This is a culture issue, not a technical one. One RCA isn’t going to change this.

Reliability is a critical facet of security from a business continuity standpoint. Any business still using crowdstrike is out of their mind.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#284
post #209

Earlier quoted context omitted.

Thank you, that's a sound perspective, but it is the responsibility of the security staff who deploy EDRs like Crowdstrike to scrub any data at ingestion time into their SIEM. but within CS's platform, it makes little sense to talk about scrubbing, since CS doesn't know what you want scrubbed unless it is standardized data forms (like SSNs,credit cards,etc..). Another way to look at it is, the CS cloud environment is…

Unfortunately the software doesn’t allow for scrubbing or redacting to be configured. Those features simply do not exist.

for their own cloud, yeah, you basically accept their cloud as an extension of your devices. but the back-end they use(d?), Splunk, does have scrubbing capability they can expose to customers, if actual customers requested it.

In reality, you can take steps to prevent PII from being logged by Crowdstrike, but credentials are too non-standard to meaningfully scrub. It would be an exercise in futility. If you trust them to have unrestricted access to the credential, the fact that they're inadvertently logging it because of the way your applications work should not be considered an increase in risk.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#285

Earlier quoted context omitted.

There's folks out there who enjoy putting out proverbial fires? I find rework like that quite frustrating

Absolutely. Some people are born firefighters. Nothing wrong with that. I once worked with a senior engineer who loved running incidents. He felt it was real engineering. He loved debugging thorny problems on a strict timeline, getting every engineer in a room and ordering them about, while also communicating widely to the company. Then, there's the rush of the all-clear and the kudos from stakeholders. Specific to h…

I agree that enjoying firefighting is not inherently harmful. However, the situation you describe afterward irks me in some way I can't quite put my finger on. A lot of words (toxic, dishonest, marketing, counterproductive, bus factor) come to mind, but none of them quite fit.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#286

I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of o…

There's folks out there who enjoy putting out proverbial fires? I find rework like that quite frustrating

I've volunteered to fight a share of fires from people who check things in untested, change infrastructure randomly, etc.

What I've learned is that fixing things for these people (and even having entire teams fixing things for weeks) just leads to a continued lax attitude to testing, and leaving the fallout for others to deal with. To them, it all worked out in the end, and they get kudos for rapidly getting a solution in place.

I'm done fixing their work. I'd rather work on my own tasks than fix all the problems with theirs. I'm strongly considering moving on, as this has become an entrenched pattern.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#287
post #23

Earlier quoted context omitted.

> If thorough investigation revealed poor quality control investment compared to what would be appropriate for a company like this, then we can say for sure. We don't really need that thorough of an investigation. They had no staged deploys when servicing millions of machines. That alone is enough to say they're not running the company correctly.

Totally agree. I’d consider staggering a rollout to be the absolute basics of due diligence. Especially when you’re building a critical part of millions of customer machines.

I also fall on the side of "stagger the rollout" (or "give customers tools to stagger the rollout"), but at the same time I recognize that a lot of customers would not accept delays on the latest malware data.

Before the incident, if you asked a customer if they would like to get updates faster even if it means that there is a remote chance of a problem with them... I bet they'd still want to get updates faster.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#288
post #234

Has anyone actually worked at a place where quality control was treated as important? I wouldn't consider this exactly surprising.

Yes, at a trading company, where important central systems had a multiweek testing process (unless the change was marked as urgent, in which case it was faster) with a dedicated team and a full replica environment which would replay historical functions 1:1 (or in some cases live), and every change needed to have an automated rollback process. Unsurprising since it directly affects the bottom line.

Very interesting. Thanks for sharing.

> every change needed to have an automated rollback process

How did you accomplish that?

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#289
post #288

Earlier quoted context omitted.

Yes, at a trading company, where important central systems had a multiweek testing process (unless the change was marked as urgent, in which case it was faster) with a dedicated team and a full replica environment which would replay historical functions 1:1 (or in some cases live), and every change needed to have an automated rollback process. Unsurprising since it directly affects the bottom line.

Very interesting. Thanks for sharing. > every change needed to have an automated rollback process How did you accomplish that?

We had a state management and deployment system through which all changes were effected that would automatically rollback changes if the smoke test failed, or if one of the ops staff found an issue.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#290
post #2

> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card. Look, I think just as poorly about CrowdStr…

Disgruntled are the Crowdstrike customers that had to deal with the outage. These employees have a lot of reputation to lose for coming forward. Crowstrike is a disgrace of a company and many others like it are doing the same behaviors but they just haven't gotten caught yet. Software development has become a disgrace when the bottom line of squeezing margins to please investors took over.

> These employees have a lot of reputation to lose for coming forward.

Employees don't typically have much reputation to ruin. I am perfectly content putting this on their shoulders.

Post reply on HN