Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

251–260 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#251

Post by pclmulqdq »

odyssey7 wrote: post #230
pclmulqdq wrote: post #86

Earlier quoted context omitted.

Shamelessly plugging my own blog post on this: https://specbranch.com/posts/knight-capital/ The TL;DR of Knight is that Knight had several things go wrong at the same time, and had no circuit breaker for the problem that did not stop trading for the whole firm for the day. Most trading firms have had things go badly, but the holes in the Swiss cheese aligned for Knight (and they were larger than many other firms). Th…

I always thought the Swiss cheese model was used to suggest that no one party could possibly be responsible for a bad thing that happened. Interesting to see the company’s culture blamed for the cheese itself.

Personally, I think there are too many things in modern American society that involve diffusion of responsibility, presumably so that people avoid negative consequences. If you're going to suggest that a system gives 1/10th of the responsibility to 10 different people, the one who made the system is the enabler of that and IMO should suffer the consequences.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#252

Post by addled »

Yesterday morning I learned that someone I was acquainted with had just passed away and the funeral is scheduled for next week.

They recently had a stroke at home just days after spending over a month in the hospital.

Then I remembered that they were originally supposed to be getting an important surgery, but it was delayed because of the CrowdStrike outage. It took weeks for the stars to align again and the surgery to happen.

It makes me wonder what the outcome would have been if they had gotten the surgery done that day, and not spent those extra weeks in the hospital with their condition and stressing about their future?

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#253

Post by skenderbeu »

Alupis wrote: post #2

> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card. Look, I think just as poorly about CrowdStr…

Disgruntled are the Crowdstrike customers that had to deal with the outage. These employees have a lot of reputation to lose for coming forward. Crowstrike is a disgrace of a company and many others like it are doing the same behaviors but they just haven't gotten caught yet. Software development has become a disgrace when the bottom line of squeezing margins to please investors took over.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#254

Post by m3047 »

bmitc wrote: post #234

Has anyone actually worked at a place where quality control was treated as important? I wouldn't consider this exactly surprising.

Yes. It was a manufacturing facility and since the products were photosensitive the entire line operated in total darkness. It was two months before they turned the lights on and I could see what I was programming for.

This was the first place I saw standups. [Edit: this was the 1990s] They were run by and for the "meat", the people running the line. "Level 2" only got to speak if we were blocked, or to briefly describe any new investigations we would be undertaking.

Weirdly (maybe?) they didn't drug test. I thought of all the places I've worked, they would. But they didn't. They were firmly committed to the "no SPOFs" doctrine and had a "tap out" policy: if anyone felt you were distracted, they could "tap you out" for the day. It was no fault. I was there for six months and three or four times I was tapped out and (after the first time, because they asked what I did with my time off the first time) told to "go climb a rock". I tapped somebody out once, for what later gossip suggested was a family issue.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#255

Post by 0xbadcafebee »

TiredOfLife wrote: post #191
0xbadcafebee wrote: post #16

Critical software infrastructure should be regulated the way critical physical infrastructure is. We don't trust the people who make buildings and bridges to "do the right thing" - we mandate it with regulations and inspections. (When your software not working strands millions of people around the globe, it's critical) And this was just a regular old "accident"; imagine the future, when a war has threat actors trying…

The regulations were the reason the companies were running Crowdstrike in the first place.

I'm saying that a (different) regulation, standard, and inspection, should apply to the whole software bill of materials, as it relates to the critical-ness of the product. Like, if security is important, the security-critical components should be inspected/tested. That's how you build a building safely: the nails are built to a certain specification and the nail vendor signs off on that.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#256

Post by insane_dreamer »

paulcole wrote: post #98
theideaofcoffee wrote: post #30

Earlier quoted context omitted.

I just don't think a company like Crowdstrike has a leg to stand on when leveling the "disgruntled" label in the face of their, let's face it, astoundingly epic fuck up. It's the disgruntled employees that I think would have the most clear picture of what was going on, regardless of them being in QA/QC or not because they, at that point, don't really care any more and will be more forthright with their thoughts. I'd…

Why would you trust a company no-man any more than a company yes-man? They both have agendas and biases. Is it just that you personally prefer one set of biases (anti-company) more than the other (pro-company)?

Well, in this case, we know one side (pro-company) fucked up big time. The other side (anti-company) may or may not have fucked up.

That makes it easier to trust one side over another.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#257

Post by insane_dreamer »

chii wrote: post #182
0xbadcafebee wrote: post #126

Earlier quoted context omitted.

I've worked in these enterprise organizations for a long time. They don't run on common sense, or even what one might consider "business sense". Their existing incentives create bizarre behavior. For example, you might think "if a big security exploit happens, the stock price might tank" . So if they value the stock price, they'll focus on security, right?. In reality what they do is focus on burying the evidence of…

While good, those ideas will all increase costs. Would you pay 10x (or more, even) for these systems? That means 10x the price of water, utilities, transport etc, which then accumulate up the chain to make other things which don't have criticality but do depend on the ones that do. The thing is, what exists today exists because it's the path of least resistence.

> Would you pay 10x (or more, even) for these systems?

if it's critical to your business, then yes; but you quickly find out whether or not it's actually critical to your business or whether it's something you can do without

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#258

Post by insane_dreamer »

bmitc wrote: post #234

Has anyone actually worked at a place where quality control was treated as important? I wouldn't consider this exactly surprising.

I haven't worked there but I would presume that systems running nuclear reactors or ICBM launchers have a strong emphasis on QC.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#259

Post by breadwinner »

ricardobayes wrote: post #223

I believe one of the biggest bad trends of the software industry as a whole is cutting down on QA/testing effort. A buggy product is almost always an unsuccessful one.

Blame Facebook and Google for that. They became successful without QA engineers, so the rest of the industry decided to follow suit in an effort to stay modern.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#260

Post by Vegenoid »

chii wrote: post #182
0xbadcafebee wrote: post #126

Earlier quoted context omitted.

I've worked in these enterprise organizations for a long time. They don't run on common sense, or even what one might consider "business sense". Their existing incentives create bizarre behavior. For example, you might think "if a big security exploit happens, the stock price might tank" . So if they value the stock price, they'll focus on security, right?. In reality what they do is focus on burying the evidence of…

While good, those ideas will all increase costs. Would you pay 10x (or more, even) for these systems? That means 10x the price of water, utilities, transport etc, which then accumulate up the chain to make other things which don't have criticality but do depend on the ones that do. The thing is, what exists today exists because it's the path of least resistence.

Consumer costs would not go up 10x to put more care into ensuring the continuous operation of critical IT infrastructure. Things like "an update to the software or configuration of critical systems must first be performed on a test system".
Post reply on HN