CrowdStrike to Delta: Stop pointing at us • Hacker News Board

Re: CrowdStrike to Delta: Stop pointing at us

#61

Post by hoistbypetard » Mon, Aug 05, 2024, 4:39 PM UTC

goosejuice wrote: post #56

hoistbypetard wrote: post #53

Earlier quoted context omitted.

There's quite some distance between "infallible" and the kind of failure mode CS's error induced. Be that as it may, I don't think the question is about infallibility, though. The question is, was the software fit for the purpose Delta used it for? And the follow-on question is whether CrowdStrike sold it for the purpose Delta used it for.

Fair, but certainly that's the responsibility of the purchaser, no? CrowdStrike couldn't feasibly know how all their customers utilize their software. That's the end user's responsibility. Software fails, machines fail, we all know this. The technical leadership at Delta should know this. Do we think software vendors should be responsible for ensuring their customers safely deploy their software? I can't imagine that…

If this were something that were being purchased off-the-shelf, it'd clearly all be the responsibility of the purchaser.

When it's priced differently depending on how the customer plans to use it, and it's sold only through a consultative process, I'm not sure it's so clear. In order to determine their pricing, the sales team demands to know how the customer is using it... so they absolutely arguably do know.

Re: CrowdStrike to Delta: Stop pointing at us

#62

Post by willhackett » Mon, Aug 05, 2024, 4:40 PM UTC

chronid wrote: post #14

willhackett wrote: post #12

Earlier quoted context omitted.

Am I right in thinking Delta could have chosen when the update was distributed to its infrastructure? In my mind, a quick test run of the update on a VM before letting it roll out globally would have revealed the BSOD boot loop.

AFAIK crowdstrike can push updates at any time at any host. There are staging areas they may use, but don't have to (particularly for definitions updates). Crowdstrike should have done a better job , but Delta chose them (to offload the responsibility and work) and now they're claiming foul. They knew the risk. This is a classic executive play of claiming the fault lies in the consultants/vendor and taking no respons…

Okay, good to know. I always thought those embedded systems would be a real pain to maintain.

Re: CrowdStrike to Delta: Stop pointing at us

#63

Post by seba_dos1 » Mon, Aug 05, 2024, 4:46 PM UTC

CrowdStrike was responsible for the first day of the outage and should deal with any damage that it caused.

Delta was responsible for the rest of its outage and should deal with any damage that it caused.

Re: CrowdStrike to Delta: Stop pointing at us

#64

Post by sjm-lbm » Mon, Aug 05, 2024, 4:51 PM UTC

paradite wrote: post #59

sjm-lbm wrote: post #25

Earlier quoted context omitted.

I really dislike this line of thinking because it assumes that Microsoft is responsible for anything you run on your Windows machine. I should be able to do whatever I want with a computer I buy, but that doesn't mean Microsoft should hold any liability for it. What am I missing here?

The difference is you are an individual and Delta is a public company. For enterprise purchases, you would have SLAs and contract terms to protect your company when something bad happens.

Fair, if this violates an SLA then Microsoft should pay up.

If Microsoft's SLAs are worded so that you're allowed to shove bits of third party code into the kernel without violating the SLA, though, they should fix that.

Re: CrowdStrike to Delta: Stop pointing at us

#65

Post by AnimalMuppet » Mon, Aug 05, 2024, 4:54 PM UTC

latentnumber wrote: post #3

> CrowdStrike said Sunday that its liability is contractually capped at an amount in the “single-digit millions.” Companies handling critical infrastructure should face more scrutiny imo.

> CrowdStrike said Sunday that its liability is contractually capped at an amount in the “single-digit millions.”

Well, that's nice. If I understand correctly, though, you can't contractually limit liability for gross negligence. I mean, you can say it in the contract, but it isn't legally enforceable.

It does raise the bar, though - gross negligence is harder to prove than ordinary negligence.

Note well: IANAL. I could be wrong.

Re: CrowdStrike to Delta: Stop pointing at us

#66

Post by latentnumber » Mon, Aug 05, 2024, 4:58 PM UTC

crngefest wrote: post #5

latentnumber wrote: post #3

> CrowdStrike said Sunday that its liability is contractually capped at an amount in the “single-digit millions.” Companies handling critical infrastructure should face more scrutiny imo.

By more „scrutiny“ do you mean increase the liability cap?

Yes, because that incentivizes such companies to be more diligent.

Re: CrowdStrike to Delta: Stop pointing at us

#67

Post by meling » Mon, Aug 05, 2024, 5:49 PM UTC

Coming from a fault tolerance background, this seems to be a prime example where OS diversity would have helped. But clearly staging the roll out of updates (even definition files) should be standard practice when you have more than 10k customers.

Re: CrowdStrike to Delta: Stop pointing at us

#68

Post by soneil » Mon, Aug 05, 2024, 7:17 PM UTC

Anon4Now wrote: post #18

One issue that hasn't received enough attention comes from a comment on Dave Plummer's video on the CrowdStrike outage. Dave Plummer is a former Windows engineer and runs a YouTube channel call Dave's Garage. @zug-zug wrote: > While this is technically what crashed machines it isn't the worst part. > CS Falcon has a way to control the staging of updates across your environment. businesses who don't want to go out of…

I'm pretty sure this is why everything we got in the first 48 hours from CS was stressing that the issue was with a "channel file" (threat definitions, content updates, etc).

Their staged update process is for the falcon driver itself. It is not for the "channel files".

As I understand it, the driver itself is understood to be a risk, and they provide facility for an N, N-1, N-2 staged deployment to mitigate this risk.

As I understand it, channel files were not identified as a risk, and were never subject to this staged deployment.

The "sell" was that you could be running a trusted driver at N-2, but still have 0day protection from up-to-date channel files. And CS's initial feedback that the issue was not with the driver itself was CYA that they hadn't been misleading customers using such staged deployments.

Re: CrowdStrike to Delta: Stop pointing at us

#69

Post by ijk » Mon, Aug 05, 2024, 7:38 PM UTC

chronid wrote: post #14

willhackett wrote: post #12

Earlier quoted context omitted.

Am I right in thinking Delta could have chosen when the update was distributed to its infrastructure? In my mind, a quick test run of the update on a VM before letting it roll out globally would have revealed the BSOD boot loop.

AFAIK crowdstrike can push updates at any time at any host. There are staging areas they may use, but don't have to (particularly for definitions updates). Crowdstrike should have done a better job , but Delta chose them (to offload the responsibility and work) and now they're claiming foul. They knew the risk. This is a classic executive play of claiming the fault lies in the consultants/vendor and taking no respons…

I'm not sure how "you should never use CrowdStrike" is an argument in CrowdStrike's favor.

I guess you're saying they shouldn't have outsourced in the first place? Which does sound like the correct conclusion in this case...

Re: CrowdStrike to Delta: Stop pointing at us

#70

Post by senectus1 » Tue, Aug 06, 2024, 12:00 AM UTC

Anon4Now wrote: post #18

One issue that hasn't received enough attention comes from a comment on Dave Plummer's video on the CrowdStrike outage. Dave Plummer is a former Windows engineer and runs a YouTube channel call Dave's Garage. @zug-zug wrote: > While this is technically what crashed machines it isn't the worst part. > CS Falcon has a way to control the staging of updates across your environment. businesses who don't want to go out of…

yeah this is bullshit, and when we spoke to our cyber dept about why we chose a product that allows this they said "all the top tier products do this".

I did suggest we turn off the proxy for the "air gapped" parts of the nextwork, and only turn it on when we're sure we're ready for it so the airgapped parts can get the updates they need. but seriously... since when is it acceptable to give a vendor control that YOU DONT HAVE over parts of your network.. crazy days.