Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

261–270 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#261

Post by odyssey7 »

pclmulqdq wrote: post #251
odyssey7 wrote: post #230

Earlier quoted context omitted.

I always thought the Swiss cheese model was used to suggest that no one party could possibly be responsible for a bad thing that happened. Interesting to see the company’s culture blamed for the cheese itself.

Personally, I think there are too many things in modern American society that involve diffusion of responsibility, presumably so that people avoid negative consequences. If you're going to suggest that a system gives 1/10th of the responsibility to 10 different people, the one who made the system is the enabler of that and IMO should suffer the consequences.

The Swiss cheese model fits better as a rebuttal when the cheese comprises both the finger-pointer and the finger-pointee. Think: sure, our software had a bug that said up was down, but what about all of your own employees who used the software, had certifications, and should have known better than to accept its conclusions?

Your usage, in assigning blame rather than diffusing it, was novel to me.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#262

Post by iudqnolq »

Alupis wrote: post #2

> “Speed was the most important thing,” said Jeff Gardner, a senior user experience designer at CrowdStrike who said he was laid off in January 2023 after two years at the company. “Quality control was not really part of our process or our conversation.” This type of article - built upon disgruntled former employees - is worth about as much as the apology GrubHub gift card. Look, I think just as poorly about CrowdStr…

There are some very specific accusations backed up by non-denials from crowdstrike.

Ex-employees said bugs caused the log monitor to drop entries. Crowdstrike responded the project was never designed to alert in real time. But Crowdstrike's website currently advertises it as working in real time.

Ex-employees said people trained to monitor laptops were assigned to monitor AWS accounts with no extra training. Crowdstrike replied that "there were no experienced ‘cloud threat hunters’ to be had" in 2022 and that optional training was available to the employees.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#263

Post by pclmulqdq »

bitcharmer wrote: post #238
0cf8612b2e1e wrote: post #56

Earlier quoted context omitted.

I do not work in finance, but surely every trading company has had an algorithm go wild at some point. Just becomes a matter of how fast someone can pull the circuit breaker before the expensive failure becomes public.

We have circuit breakers for that very purpose. Everyone on the street does. It's just that theirs seems to have failed for some reason.

Theirs didn't fail, and they did have one. The circuit breaker they had that would have worked was a big red button that killed all of their trading processes, which would have meant spending the rest of the day figuring out and unwinding their positions.

Ihey were unwilling to push that button in the short time they had. If you read the reports to the SEC or the articles about it, you will note that. The follow-ups recommended that all firms adopt a big red button that is less catastrophic.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#264

Post by tsimionescu »

mewpmewp2 wrote: post #203
tsimionescu wrote: post #172

Earlier quoted context omitted.

Not sure, but definitely more enterprisey than "release a patch to the entire world at once before running it on a single machine in-house".

So it would be preferable to have your data encrypted, taken hostage unless you pay, and be down for days, instead of 6 hours of just down?

Do you seriously believe that all CrowdStrike on Windows customers were at such imminent risk of ransomware that one-two hours to run this on one internal setup and catch the critical error they released would have been dangerous?

This is a ludicrous position, and has been proven obviously false by the proceedings: all systems that were crashed by this critical failure were not, in fact, attacked with ransomware once the CS agent was un-installed (at great pain).

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#265

Post by oooyay »

hitekker wrote: post #248
wesselbindt wrote: post #241

Earlier quoted context omitted.

There's folks out there who enjoy putting out proverbial fires? I find rework like that quite frustrating

Absolutely. Some people are born firefighters. Nothing wrong with that. I once worked with a senior engineer who loved running incidents. He felt it was real engineering. He loved debugging thorny problems on a strict timeline, getting every engineer in a room and ordering them about, while also communicating widely to the company. Then, there's the rush of the all-clear and the kudos from stakeholders. Specific to h…

That's called hero culture and there's definitely something wrong with it.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#266

Post by oehpr »

addled wrote: post #252

Yesterday morning I learned that someone I was acquainted with had just passed away and the funeral is scheduled for next week. They recently had a stroke at home just days after spending over a month in the hospital. Then I remembered that they were originally supposed to be getting an important surgery, but it was delayed because of the CrowdStrike outage. It took weeks for the stars to align again and the surgery…

I appreciate your post here and I'm glad you shared, because it's an example of a distributed harm. One of millions to shake out of this incident, that doesn't have a dollar figure, so it doesn't really "count".

To illustrate:

If I were to do something horrible like kick a 3 year olds knee out and cripple them for life, I would be rightly labeled a monster.

But If I were to say... advocate for education reform to push American Sign Language out of schools, so that deaf children grow up without a developmental language? We don't have words for that, and if we did, none of them would get near the cumulative scope and harm of that act.

We simply do not address distributed harms correctly. And a big part of it is that we don't, we can't, see all the tangible harms it causes.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#267

Post by namdnay »

addled wrote: post #252

Yesterday morning I learned that someone I was acquainted with had just passed away and the funeral is scheduled for next week. They recently had a stroke at home just days after spending over a month in the hospital. Then I remembered that they were originally supposed to be getting an important surgery, but it was delayed because of the CrowdStrike outage. It took weeks for the stars to align again and the surgery…

Not to defend Crowdstrike in any way, but it’s a bit unfair to only look at the downside. What if his hospital hadn’t bought an antivirus, and got hit by ransomware?

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#268

Post by jamesmotherway »

wesselbindt wrote: post #241
hitekker wrote: post #239

I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of o…

There's folks out there who enjoy putting out proverbial fires? I find rework like that quite frustrating

Some people rise to the occasion during crises and find it rewarding. There's a lot of pop science around COMT (the "warrior gene" associated with stress resilience), which I take with a grain of salt. There does seem to be something there, though, and it overlaps with my personal experience that many great security operations people tend to have ADHD traits.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#269

Post by Izkata »

GuB-42 wrote: post #222
hn_throwaway_99 wrote: post #70

Earlier quoted context omitted.

While I agree with this, from a software engineering perspective I think it's more useful to look at the lessons learned. I think it's too easy to just throw "Crowdstrike is a bunch of idiots" against the wall, and I don't think that's true. It's clear to me that CrowdStrike saw this as a data update vs. a code update, and that they had much more stringent QA procedures for code updates that they did data updates. It…

It could have been ok to expedite data updates, should the code treat configuration data as untrusted input, as if it could be written by an attacker. It means fuzz testing and all that. Obviously the system wasn't very robust, as a simple, within specs change could break it. A company like CrowdStrike, which routinely deals with memory exploits and claims to do "zero trust" should know better. As often, there is a g…

> Obviously the system wasn't very robust, as a simple, within specs change could break it.

From my limited understanding, the file was corrupted in some way. Lots of NULL bytes, something like that.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#270

Post by acdha »

hello_moto wrote: post #154
acdha wrote: post #89

Earlier quoted context omitted.

They probably weren’t, but that still speaks to their general culture and is compatible with what we know about their kernel engineering culture (limited testing, no review, no use of common fail safe mechanisms).

A company can have different business units with different culture/mentality. I bet my ass anyone working in low-level code don't ship the way you do in Cloud.

> I bet my ass anyone working in low-level code don't ship the way you do in Cloud.

Their technical report says otherwise – and we know they didn’t adopt the common cloud practices of doing real testing before shipping or having a progressive deployment.

Post reply on HN