Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

231–240 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#231
post #2

> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card. Look, I think just as poorly about CrowdStr…

I do agree with having to expect bias there, but who else do you really expect to speak out?Any current employee would very quickly become an ex-employee if they speak out with any specifics.

I would expect any contractor that may have worked for CrowdStrike, or done something like a third-party audit, would be under an NDA covering their work.

Who's left to speak out with any meaningful details?

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#233

Earlier quoted context omitted.

It's not about the blame, it's about how you respond to incidents and what mitigation steps you take. Even if they aren't directly responsible, they clearly didn't take proper mitigation steps when they encountered the problem.

How do you mitigate the OS breaking an API below you in an update? Test the updates before they come out? Even if you could, you'd still need to deploy a fix before the OS update hits the customers, and anyone that didn't update would still be affected. The linux case is just _very_ different from the windows case. The mitigation steps that could have been taken to avoid the linux problem would not have helped for th…

You're missing the forest for the trees.

It's: a) an update, b) pushed out globally without proper testing, c) that bricked the OS.

It's an obvious failure mode that if you have a proper incident response process would be revealed from that specific incident and flagged for needing mitigation.

I do this specific thing for a living. You don't just address the exact failure that happened but try to identify classes of risk in your platform.

> Even if you could, you'd still need to deploy a fix before the OS update hits the customers, and anyone that didn't update would still be affected.

And yet the problem would still only affect Crowdstrike's paying customers. No matter how much you blame upstream your paying customers are only ever going to blame their vendor because the vendor had discretion to test and not release the update. As their customers should.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#235
post #234

Has anyone actually worked at a place where quality control was treated as important? I wouldn't consider this exactly surprising.

Nope. Did everyone forget the tech motto "move fast and break things"? Where is the room for quality control in that philosophy?

Corps won't even put resource into anti-fraud efforts if they believe the millions being stolen from their bottom line isn't worth the effort. I have seen this attitude working in FAANGS.

None of this will change until tech workers stop being sadists and actually unionize.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#237
post #2

> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card. Look, I think just as poorly about CrowdStr…

> Quality control was not really part of our process or our conversation.

Is anyone really surprised or learned any new information? For us that have worked for tech companies, this is one of those repeating complaints that you hear across orgs that indicates a less than stellar engineering culture.

I've worked with numerous F500 orgs and I would say 3/5 orgs that I worked in, their code was so bad that it made me wonder how they haven't had a major incident yet.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#238

Earlier quoted context omitted.

Knight Capitol: "$10 million a minute. That’s about how much the trading problem that set off turmoil on the stock market on Wednesday morning is already costing the trading firm. The Knight Capital Group announced on Thursday that it lost $440 million when it sold all the stocks it accidentally bought Wednesday morning because a computer glitch. " Glitch. Oh... https://en.wikipedia.org/wiki/Therac-25

I do not work in finance, but surely every trading company has had an algorithm go wild at some point. Just becomes a matter of how fast someone can pull the circuit breaker before the expensive failure becomes public.

We have circuit breakers for that very purpose. Everyone on the street does. It's just that theirs seems to have failed for some reason.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#239
I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of our process'.

My guess is that people are identifying with sentence said just before: "Speed [of shipping] is everything." Aka "Move fast and break things."

The culture described by this article must mirror many of our lived experiences. The pure pleasure of shipping code, putting out fires, making an impact (positive or negative)... and then leaving it to the next engineers & managers to sort out, ignoring the mess until it explodes. Even when it does, no one gets blamed for the outage and soon everyone goes back to building features that get them promoted, regardless of quality.

Through that ZIRP light, these process failures must look like a feature, not a bug. The emphasis on "quality" must also look like annoying roadblocks in the way of having fun on the customer's dime.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#240

anyone feel like this and Boeing sound remarkably similar? Its almost like there is a lesson for executives here. hmmmm

The only lesson for these people is loss of bonuses. This will keep happening for as long as golden parachutes are a thing.
Post reply on HN