Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

61–70 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#61

Post by worik »

0cf8612b2e1e wrote: post #56
ForOldHack wrote: post #45

Earlier quoted context omitted.

Knight Capitol: "$10 million a minute. That’s about how much the trading problem that set off turmoil on the stock market on Wednesday morning is already costing the trading firm. The Knight Capital Group announced on Thursday that it lost $440 million when it sold all the stocks it accidentally bought Wednesday morning because a computer glitch. " Glitch. Oh... https://en.wikipedia.org/wiki/Therac-25

I do not work in finance, but surely every trading company has had an algorithm go wild at some point. Just becomes a matter of how fast someone can pull the circuit breaker before the expensive failure becomes public.

> surely every trading company has had an algorithm go wild at some point.

You would think so.

Cynical me.

But no. When money is at stake much more care is taken than when lives are at stake.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#63

Post by nullvoxpopuli »

bb88 wrote: post #20

Most interesting quote in the article: “It was hard to get people to do sufficient testing sometimes,” said Preston Sego, who worked at CrowdStrike from 2019 to 2023. His job was to review the tests completed by user experience developers that alerted engineers to bugs before proposed coding changes were released to customers. Sego said he was fired in February 2023 as an “insider threat” after he criticized the comp…

I'd like to clarify: that my job was also to educate, modernize, and improve developer velocity through tooling and framework updates / changes (impacting every team in my department (UX / frontend engineering)).

Reviewing tests is part of PR review.

--- and before anyone asks, this is my statement on CrowdStrike calling everyone disgruntled:

"I'm not disgruntled.

But as a shareholder (and probably more primarily, someone who cares about coworkers), I am disappointed.

For the most part, I'm still mourning the loss of working with the UX/Platform team."

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#65

Post by hyperpape »

panic wrote: post #48

Why would it matter? The absolute worst case scenario happened and their stock is still up 50% YoY, beating the S&P 500.

It’s down 30% since the incident, and flat since 3 years ago.

If it runs up a huge amount in the first half of the year and then the incident knocks off 30% of their market, that still means the incident was really bad.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#66

Post by neverrroot »

taspeotis wrote: post #52
goralph wrote: post #51

What are some alternatives to CrowdStrike?

Personal: Nothing - Windows Defender is built into Windows. Business: Nothing - Windows Defender Advanced Threat Protection is built into the higher Microsoft 365 license tiers. It amazes me people chose to pay money to have all their PCs bluescreen.

This is a good example of very limited thinking.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#67

Post by neverrroot »

dgfitz wrote: post #29
nine_zeros wrote: post #14

Typical of tech companies these days. Quality is considered immaterial - or worse - put on low level managers and engineers who don't have the time to clearly examine quality and good roll out practices. C-Suite and investors don't seem to want to spend on quality. They should just price in that their stock investment could collapse any day.

[flagged]

This ^^

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#68

Post by [deleted] »

apimade wrote: post #43
st3fan wrote: post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

Is this really a criticism? Because this has been the case forever with all security and SIEM tools. It’s one of the reasons why the SIEM is the most locked down pieces of software in the business. Realistically, secrets alone shouldn’t allow an attacker access - they should need access to infrastructure or a certificates in machines as well. But unfortunately that’s not the case for many SaaS vendors.

[deleted]

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#69

Post by chelmzy »

apimade wrote: post #43
st3fan wrote: post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

Is this really a criticism? Because this has been the case forever with all security and SIEM tools. It’s one of the reasons why the SIEM is the most locked down pieces of software in the business. Realistically, secrets alone shouldn’t allow an attacker access - they should need access to infrastructure or a certificates in machines as well. But unfortunately that’s not the case for many SaaS vendors.

Most sane SIEM engineers would implement masking for this. Not sure if CS still uses Splunk but they did at one point. No excuse really.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#70

Post by hn_throwaway_99 »

daedrdev wrote: post #8
mewpmewp2 wrote: post #5

Earlier quoted context omitted.

There definitely was a huge outage, but based on the given information we still can't know for sure how much they invested in testing and quality control. There's always a chance of failure even for the most meticulous companies. Now I'm not defending or excusing the company, but a singular event like this can happen to anyone and nothing is 100%. If thorough investigation revealed poor quality control investment com…

Two things are clear though Nobody ran this update The update was pushed globally to all computers With that alone we know they have failed the simplest of quality control methods for a piece of software as widespread as theirs. This is even excluding that there should have been some kind of error handling to allow the computer to boot if they did push bad code.

While I agree with this, from a software engineering perspective I think it's more useful to look at the lessons learned. I think it's too easy to just throw "Crowdstrike is a bunch of idiots" against the wall, and I don't think that's true.

It's clear to me that CrowdStrike saw this as a data update vs. a code update, and that they had much more stringent QA procedures for code updates that they did data updates. It's very easy for organizations to lull themselves into this false sense of security when they make these kinds of delineations (sometimes even subconsciously at first), and then over time they lose site of the fact that a bad data update can be just as catastrophic as a bad code update. I've seen shades of this issue elsewhere many times.

So all that said, I think your point is valid. I know Crowdstrike had the posture that they wanted to get vulnerability files deployed globally as fast as possible upon a new threat detection in order to protect their clients, but it wouldn't have been that hard to build in some simple checks in their build process (first deploy to a test bed, then deploy globally) even if they felt a slower staged rollout would have left too many of their clients unprotected for too long.

Hindsight is always 20/20, but I think the most important lesson is that this code vs data dichotomy can be dangerous if the implications are not fully understood.

Post reply on HN