Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

241–250 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#241

Post by wesselbindt »

hitekker wrote: post #239

I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of o…

There's folks out there who enjoy putting out proverbial fires? I find rework like that quite frustrating

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#242

Post by wesselbindt »

sersi wrote: post #228

Crowdstrike was heavily pushed on us at a previous company both for compliance reason by some of our clients (BCG were the ones pushing us to use crowdstrike) and from our liability insurance company. It was really an uphill battle to convince everyone not to use Crowdstrike. Eventually I managed to but after many meetings where I had to spend a significant amount of time convincing different shareholders. I'm sure a…

What made you unwilling to use CS at the time?

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#243

Post by zipy124 »

tw04 wrote: post #12
sonofhans wrote: post #7

Earlier quoted context omitted.

If design isn’t involved in QC you’re not doing QC very well. If design isn’t plugged into development process enough to understand QC then you’re not doing design very well.

Why would a UX designer be involved in any way, shape, or form in kernel level code patches? They would literally never ship an update if they had that many hands in the pot for something completely unrelated. Should they also have their sales reps and marketing folks pre-brief before they make any code changes?

I would agree if it was a UI designer, but a good UX designer designs for the users, which in this case including the system admins who will be updating kernel level code patches. Ensuring they have a good experience e.g no crashes, is their job. A recommendation would likely be for example small roll-outs to minimise the number of people having a bad user experience on a roll-out that goes wrong.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#244

Post by wesselbindt »

bb88 wrote: post #20

Most interesting quote in the article: “It was hard to get people to do sufficient testing sometimes,” said Preston Sego, who worked at CrowdStrike from 2019 to 2023. His job was to review the tests completed by user experience developers that alerted engineers to bugs before proposed coding changes were released to customers. Sego said he was fired in February 2023 as an “insider threat” after he criticized the comp…

> return to work

I know you're just quoting the phrase, but what a gross and dishonest way of phrasing "return to office". Implies working remotely doesn't count as work. Smacks of PR. Yuck.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#245

Post by wesselbindt »

bitcharmer wrote: post #240
manvillej wrote: post #232

anyone feel like this and Boeing sound remarkably similar? Its almost like there is a lesson for executives here. hmmmm

The only lesson for these people is loss of bonuses. This will keep happening for as long as golden parachutes are a thing.

How can we get rid of golden parachutes?

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#246

Post by MichaelZuo »

wesselbindt wrote: post #241
hitekker wrote: post #239

I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of o…

There's folks out there who enjoy putting out proverbial fires? I find rework like that quite frustrating

Well there are a handful of expert consultants who do, since they charge an eye watering price per hour for putting out fires.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#247

Post by xeromal »

mewpmewp2 wrote: post #203
tsimionescu wrote: post #172

Earlier quoted context omitted.

Not sure, but definitely more enterprisey than "release a patch to the entire world at once before running it on a single machine in-house".

So it would be preferable to have your data encrypted, taken hostage unless you pay, and be down for days, instead of 6 hours of just down?

That's a false dichotomy

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#248

Post by hitekker »

wesselbindt wrote: post #241
hitekker wrote: post #239

I was surprised by how dismissive these comments are. Former staff members, engineers included, are claiming that their former company's unsafe development culture contributed to a colossal world-wide outage & other previous outages. These employee's allegations ought to be seen as credible, or at least as informative. Instead, many seem to be attacking the UX designer commenting on 'Quality control was not part of o…

There's folks out there who enjoy putting out proverbial fires? I find rework like that quite frustrating

Absolutely. Some people are born firefighters. Nothing wrong with that.

I once worked with a senior engineer who loved running incidents. He felt it was real engineering. He loved debugging thorny problems on a strict timeline, getting every engineer in a room and ordering them about, while also communicating widely to the company. Then, there's the rush of the all-clear and the kudos from stakeholders.

Specific to his situation, I think he enjoyed the inflated ownership that the sudden urgency demanded. The system we owned was largely taken for granted by the org; a dead-end for a career. Calling incidents was a good way to get visibility at low-cost, i.e., no one would follow-up on our postmortem action items.

It eventually became a problem, though, when the system we owned was essentially put into maintenance mode, aka zero development velocity. Then I estimate (balancing for other variables) the rate the senior engineer called an incident for not-incidents went up by 3x...

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#249

Post by RaftPeople »

hn_throwaway_99 wrote: post #70
daedrdev wrote: post #8

Earlier quoted context omitted.

Two things are clear though Nobody ran this update The update was pushed globally to all computers With that alone we know they have failed the simplest of quality control methods for a piece of software as widespread as theirs. This is even excluding that there should have been some kind of error handling to allow the computer to boot if they did push bad code.

While I agree with this, from a software engineering perspective I think it's more useful to look at the lessons learned. I think it's too easy to just throw "Crowdstrike is a bunch of idiots" against the wall, and I don't think that's true. It's clear to me that CrowdStrike saw this as a data update vs. a code update, and that they had much more stringent QA procedures for code updates that they did data updates. It…

> It's clear to me that CrowdStrike saw this as a data update vs. a code update

> Hindsight is always 20/20, but I think the most important lesson is that this code vs data dichotomy can be dangerous if the implications are not fully understood.

But it's not some new condition that the industry hasn't already been dealing with for many many decades (i.e. code vs config vs data vs any other type of change to system, etc.).

There are known strategies to reduce the risk.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#250

Post by sudosysgen »

bmitc wrote: post #234

Has anyone actually worked at a place where quality control was treated as important? I wouldn't consider this exactly surprising.

Yes, at a trading company, where important central systems had a multiweek testing process (unless the change was marked as urgent, in which case it was faster) with a dedicated team and a full replica environment which would replay historical functions 1:1 (or in some cases live), and every change needed to have an automated rollback process. Unsurprising since it directly affects the bottom line.
Post reply on HN