Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

291–300 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#291

Post by Aeolun »

cj wrote: post #131
stogot wrote: post #118

Earlier quoted context omitted.

Exporting to a SIEM does not correlate to either of those extremes. It’s stupidity and makes auditing worse

SIEM = Security Information & Event Management Factually, it is necessary for auditing and absolutely correlates with the extreme of needing to monitor the “usage” of “secrets”. In a highly auditable/“secure” environment, you can’t give secrets to employees with no tracking of when the secrets are used.

> In a highly auditable/“secure” environment, you can’t give secrets to employees with no tracking of when the secrets are used.

Yeah. So you track them when they are used (which also gives you a nice timestamp). Not when they’re just sitting in the env.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#292

Post by addled »

namdnay wrote: post #267
addled wrote: post #252

Yesterday morning I learned that someone I was acquainted with had just passed away and the funeral is scheduled for next week. They recently had a stroke at home just days after spending over a month in the hospital. Then I remembered that they were originally supposed to be getting an important surgery, but it was delayed because of the CrowdStrike outage. It took weeks for the stars to align again and the surgery…

Not to defend Crowdstrike in any way, but it’s a bit unfair to only look at the downside. What if his hospital hadn’t bought an antivirus, and got hit by ransomware?

Sure, and even if the surgery happened on time, they still might have had a stroke once they got home and had the same outcome.

But as other posts on HN have discussed, anecdotes, especially your own, hit differently.

It makes me thankful the software I work on isn't involved in life and death situations... But then again, it causes me to better consider the things my work could be responsible for (banking). Rushed work that causes a loan application to fail or transaction to be held unnecessarily shouldn't kill someone outright, but there can be real consequences that affect real people just like Rita.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#294

Post by MichaelRo »

This "security" thing is getting ridiculous. It's become the Gestapo of information technology, they can do anything they want when they want to your computer, cannot resist it and there's absolutely no transparency on what they do to you and why.

I've recently changed jobs and the new employer, a large company, obviously has to have an IT compliance / security update policy because everyone else has it so if they stand out from the crowd and don't do it and somehow get hacked, it's 100x worse than constantly annoying employees and top of the line computers working like a 1970s terminal.

It's rarely that a week passes without the obligatory update + restart. And at least once a month they update THE FUCKING BIOS! What the fuck can be so broken in those laptops that the BIOS is a constant security hazard?! And why would you buy software from someone who week after week after week tell you all you had so far was a hazardous piece of shit that cannot possibly function without constant pampering?

Ahh and of course they botch it. Had to have the OS completely wiped out and reinstalled after the laptop started to behave more and more erratically, 100% caused by faulty updates on top of faulty patches trying to patch the faulty updates. Worked OK for a while afterwards then updates started piling up and so far I only lost use of the web camera (before it was Wifi then display adapter).

There's literally no words how much I hate "the system" and the constant security update take it up the ass we're forced to put up with.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#295

Post by hulitu »

mewpmewp2 wrote: post #5
nyc_data_geek1 wrote: post #4

Earlier quoted context omitted.

>>So basically we have nothing. Except the biggest IT outage ever. And a postmortem showing their validation checks were insufficient. And a rollout process that did not stage at all, just rawdogged straight to global prod. And no lab where the new code was actually installed and run prior to global rawdogging. I'd say there's smoke, and numerous accounts of fire, which this can be taken in the context of.

There definitely was a huge outage, but based on the given information we still can't know for sure how much they invested in testing and quality control. There's always a chance of failure even for the most meticulous companies. Now I'm not defending or excusing the company, but a singular event like this can happen to anyone and nothing is 100%. If thorough investigation revealed poor quality control investment com…

[flagged]

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#296

Post by bmitc »

m3047 wrote: post #254
bmitc wrote: post #234

Has anyone actually worked at a place where quality control was treated as important? I wouldn't consider this exactly surprising.

Yes. It was a manufacturing facility and since the products were photosensitive the entire line operated in total darkness. It was two months before they turned the lights on and I could see what I was programming for. This was the first place I saw standups. [Edit: this was the 1990s] They were run by and for the "meat", the people running the line. "Level 2" only got to speak if we were blocked, or to briefly descr…

That sounds ... intense, to say the least.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#297

Post by m3047 »

bmitc wrote: post #296
m3047 wrote: post #254

Earlier quoted context omitted.

Yes. It was a manufacturing facility and since the products were photosensitive the entire line operated in total darkness. It was two months before they turned the lights on and I could see what I was programming for. This was the first place I saw standups. [Edit: this was the 1990s] They were run by and for the "meat", the people running the line. "Level 2" only got to speak if we were blocked, or to briefly descr…

That sounds ... intense, to say the least.

It was a machine. At first it was kind of creepy to have the feeling that when you entered the building you were part of a machine. But after a couple of weeks it was addictive and I have never looked forward to going to work somewhere as much as I did while working there. Even climbing the rocks on my enforced days off gained a mental narrative that "I'm climbing this rock to be the best part of the machine I can be".

Sure most of the times I was tapped out I was distracted by personal thoughts. But one time I was just thinking about the problem. I protested "but I was thinking about the problem!" and they said "go think somewhere else!".

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#298

Post by dartos »

wlonkly wrote: post #287
dartos wrote: post #23

Earlier quoted context omitted.

Totally agree. I’d consider staggering a rollout to be the absolute basics of due diligence. Especially when you’re building a critical part of millions of customer machines.

I also fall on the side of "stagger the rollout" (or "give customers tools to stagger the rollout"), but at the same time I recognize that a lot of customers would not accept delays on the latest malware data. Before the incident, if you asked a customer if they would like to get updates faster even if it means that there is a remote chance of a problem with them... I bet they'd still want to get updates faster.

There must be balance

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#299

Post by hitekker »

wesselbindt wrote: post #285
hitekker wrote: post #248

Earlier quoted context omitted.

Absolutely. Some people are born firefighters. Nothing wrong with that. I once worked with a senior engineer who loved running incidents. He felt it was real engineering. He loved debugging thorny problems on a strict timeline, getting every engineer in a room and ordering them about, while also communicating widely to the company. Then, there's the rush of the all-clear and the kudos from stakeholders. Specific to h…

I agree that enjoying firefighting is not inherently harmful. However, the situation you describe afterward irks me in some way I can't quite put my finger on. A lot of words (toxic, dishonest, marketing, counterproductive, bus factor) come to mind, but none of them quite fit.

I have a word in mind, but I'll save it for a blogpost one day.

To be fair to the senior, it was a bad situation made worse by everyone's self-interest. Myself included.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#300

Post by NeutralCrane »

namdnay wrote: post #267
addled wrote: post #252

Yesterday morning I learned that someone I was acquainted with had just passed away and the funeral is scheduled for next week. They recently had a stroke at home just days after spending over a month in the hospital. Then I remembered that they were originally supposed to be getting an important surgery, but it was delayed because of the CrowdStrike outage. It took weeks for the stars to align again and the surgery…

Not to defend Crowdstrike in any way, but it’s a bit unfair to only look at the downside. What if his hospital hadn’t bought an antivirus, and got hit by ransomware?

Who needs ransomware when your antivirus platform is perfectly capable of bricking your computer systems?
Post reply on HN