Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike ex-employees: 'Quality control was not part of our process'

semafor.com

Post reply on HN
Print view View original

31–40 of 311 posts

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#31

Post by mewpmewp2 »

dartos wrote: post #23
idkwhatimdoin wrote: post #18

Earlier quoted context omitted.

> If thorough investigation revealed poor quality control investment compared to what would be appropriate for a company like this, then we can say for sure. We don't really need that thorough of an investigation. They had no staged deploys when servicing millions of machines. That alone is enough to say they're not running the company correctly.

Totally agree. I’d consider staggering a rollout to be the absolute basics of due diligence. Especially when you’re building a critical part of millions of customer machines.

I would say that canary release is an absolute must 100%. Except I can think of cases where it might still not be enough. So, I just don't feel comfortable judging them out of the box. Does all the evidence seem to point against them? For sure. But I just don't feel comfortable giving that final verdict without knowing for sure.

Specifically because this is about fighting against malicious actors, where time can be of essence to deploy some sort of protection against a novel threat.

If there's deadlines that you can go over, and nothing bad happens, for sure. Always have canary releases, and perfect QA, monitoring everything thoroughly, but I'm just saying, there can be cases where damage that could be done if you don't act fast enough, is just so much worse.

And I don't know that it wasn't the case for them. I just don't know.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#33

Post by dartos »

mewpmewp2 wrote: post #31
dartos wrote: post #23

Earlier quoted context omitted.

Totally agree. I’d consider staggering a rollout to be the absolute basics of due diligence. Especially when you’re building a critical part of millions of customer machines.

I would say that canary release is an absolute must 100%. Except I can think of cases where it might still not be enough. So, I just don't feel comfortable judging them out of the box. Does all the evidence seem to point against them? For sure. But I just don't feel comfortable giving that final verdict without knowing for sure. Specifically because this is about fighting against malicious actors, where time can be o…

In this case, they pretty much caused a worst case scenario…

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#34

Post by janalsncm »

theideaofcoffee wrote: post #25
0xbadcafebee wrote: post #16

Critical software infrastructure should be regulated the way critical physical infrastructure is. We don't trust the people who make buildings and bridges to "do the right thing" - we mandate it with regulations and inspections. (When your software not working strands millions of people around the globe, it's critical) And this was just a regular old "accident"; imagine the future, when a war has threat actors trying…

"We can't regulate the industry because then the US loses to China" or "regulation will kill the US competitive advantage!" responses I've had to suggesting the same and I just can't. But I agree with you 100%. If it's safety critical, it should be under even more scrutiny than other things, it shouldn't be left to self-regulating QA-like processes in profit seeking companies and has to have a bit more scrutiny befor…

> then the US loses to China

Yeah it makes no sense. Was the US not losing to China when we own-goaled the biggest cybersecurity incident in history?

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#35

Post by Zigurd »

theideaofcoffee wrote: post #25
0xbadcafebee wrote: post #16

Critical software infrastructure should be regulated the way critical physical infrastructure is. We don't trust the people who make buildings and bridges to "do the right thing" - we mandate it with regulations and inspections. (When your software not working strands millions of people around the globe, it's critical) And this was just a regular old "accident"; imagine the future, when a war has threat actors trying…

"We can't regulate the industry because then the US loses to China" or "regulation will kill the US competitive advantage!" responses I've had to suggesting the same and I just can't. But I agree with you 100%. If it's safety critical, it should be under even more scrutiny than other things, it shouldn't be left to self-regulating QA-like processes in profit seeking companies and has to have a bit more scrutiny befor…

Not to mention humans going extinct because regulators are to blame for there being no city on Mars. Because that's definitely the reason there's no city on Mars.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#36

Post by st3fan »

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text.

Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs.

Only the Mac version does this. There is no way to disable this behaviour or a way to redact things.

Another really odd design decision. They probably have many many thousands of plain text secrets from their customers stored in their SIEM.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#37

Post by x3n0ph3n3 »

st3fan wrote: post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

Can you provide some more info on this? How do you know? Is this documented somewhere?

I'm sure this is going to raise red-flags in my IT department.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#38

Post by Zigurd »

insane_dreamer wrote: post #26

> CrowdStrike disputed much of Semafor’s reporting I expect some ex-employees to be disgruntled and present things in a way that makes CroudStrike look bad. That happens with every company. BUT, CrowdStrike has ZERO credibility at this point. I don't believe a word they say.

At some companies, like Boeing, the shorter list would be the gruntled employees.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#39

Post by st3fan »

x3n0ph3n3 wrote: post #37
st3fan wrote: post #36

Found out that the CrowdStrike Mac agent (Falcon) sends all your secrets from environment variables to their cloud hosted SIEM. In plain text. Anyone with access to your CS SIEM can search for GitHub, aws, etc creds. Anything your devs, ops and sec teams use on their Macs. Only the Mac version does this. There is no way to disable this behaviour or a way to redact things. Another really odd design decision. They prob…

Can you provide some more info on this? How do you know? Is this documented somewhere? I'm sure this is going to raise red-flags in my IT department.

Ask them to search for the usual env var names like GITHUB_TOKEN or AWS_ACCESS_KEY_ID.

Re: CrowdStrike ex-employees: 'Quality control was not part of our process'

#40

Post by kccqzy »

sundvor wrote: post #24
nyc_data_geek1 wrote: post #4

Earlier quoted context omitted.

>>So basically we have nothing. Except the biggest IT outage ever. And a postmortem showing their validation checks were insufficient. And a rollout process that did not stage at all, just rawdogged straight to global prod. And no lab where the new code was actually installed and run prior to global rawdogging. I'd say there's smoke, and numerous accounts of fire, which this can be taken in the context of.

"Everyone" piles on Tesla all the time; a worthwhile comparison would be how Tesla roll out vehicle updates. Sometimes people are up in arms "where's my next version" (eg when adaptive headlights was introduced), yet Tesla prioritise a safe, slow roll out. Sometimes the updates fail (and get resolved individually), but never on a global scale. (None experienced myself, as a TM3 owner on the "advanced" update preferen…

You can also say the same thing about Google. Just go look at the release notes on the App Store for the Google Home app. There was a period of more than six months where every single release said "over the next few weeks we're rolling out the totally redesigned Google Home app: new easier to navigate 5-tab layout."

When I read the same release notes so often I begin to question whether this redesign is really taking more than six months to roll out. And then I read the Sonos app disaster and I thought that was the other extreme.

Post reply on HN