I've been at companies where we would do a bunch of processing overnight. One time, CPU died on a server in the middle of processing. Apparently we didn't have procedure for recovering in the middle of a process. Took us two days to write code to recover and we had to contact a third party vendor to get "What data did we send you?"
CrowdStrike to Delta: Stop pointing at us
11–20 of 76 posts
Re: CrowdStrike to Delta: Stop pointing at us
#12> CrowdStrike said Sunday that its liability is contractually capped at an amount in the “single-digit millions.” Companies handling critical infrastructure should face more scrutiny imo.
Crowdstrike is not handling critical infrastructure. Delta is. The reality is the industry wants its cake and eat it too. No one forced Delta to buy a software which could force upgrades in their production fleet. They're a billion dollars company, and should put their big boys pants on.
In my mind, a quick test run of the update on a VM before letting it roll out globally would have revealed the BSOD boot loop.
Re: CrowdStrike to Delta: Stop pointing at us
#13Earlier quoted context omitted.
Crowdstrike is not handling critical infrastructure. Delta is. The reality is the industry wants its cake and eat it too. No one forced Delta to buy a software which could force upgrades in their production fleet. They're a billion dollars company, and should put their big boys pants on.
Am I right in thinking Delta could have chosen when the update was distributed to its infrastructure? In my mind, a quick test run of the update on a VM before letting it roll out globally would have revealed the BSOD boot loop.
Re: CrowdStrike to Delta: Stop pointing at us
#14Earlier quoted context omitted.
Crowdstrike is not handling critical infrastructure. Delta is. The reality is the industry wants its cake and eat it too. No one forced Delta to buy a software which could force upgrades in their production fleet. They're a billion dollars company, and should put their big boys pants on.
Am I right in thinking Delta could have chosen when the update was distributed to its infrastructure? In my mind, a quick test run of the update on a VM before letting it roll out globally would have revealed the BSOD boot loop.
Crowdstrike should have done a better job, but Delta chose them (to offload the responsibility and work) and now they're claiming foul. They knew the risk. This is a classic executive play of claiming the fault lies in the consultants/vendor and taking no responsibility.
Re: CrowdStrike to Delta: Stop pointing at us
#15I struggle to understand MSFTs liability here. Can anyone explain to me how Microsoft would be liable for Delta’s outage?
This just sounds like shifting the blame to me. Nothing Microsoft did changed anything for the CS situation. Apparently it's Microsoft's fault that they don't have a great, secure, bugless API that everyone seems to want all of the sudden.
On the day the crashes started, Microsoft had a cloud services outage, and people assumed the two outages were related (they sort of were, in that some of the consultancies Microsoft hired also ran CrowdStrike so they couldn't get support for the cloud outages as easily). That seemed to have stuck around.
Re: CrowdStrike to Delta: Stop pointing at us
#16Am I reading this right that this amounts to "you decided yourself to install our software on your devices, you should have known better?"
Re: CrowdStrike to Delta: Stop pointing at us
#17I struggle to understand MSFTs liability here. Can anyone explain to me how Microsoft would be liable for Delta’s outage?
Not providing a way to inspect those data except from within kernel drivers (or whatever Windows calls them.) The huge HN thread about what happened weeks ago had some comments about Linux using eBPF to get the same kind of information Crowdstrike needs and Macs having another technology to do the same thing. In both cases the kernel won't crash and take down the machine. Of course it's possible to hog a machine from…
Re: CrowdStrike to Delta: Stop pointing at us
#18@zug-zug wrote:
> While this is technically what crashed machines it isn't the worst part.
> CS Falcon has a way to control the staging of updates across your environment. businesses who don't want to go out of business have a N-1 or greater staging policy and only test systems get the latest updates immediately. My work for example has a test group at N staging, a small group of noncritical systems at N-1, and the rest of our computers at N-2.
> This broken update IGNORED our staging policies and went to ALL machine at the same time. CS informed us after our business was brought down that this is by design and some updates bypass policies.
> So in the end, CS caused untold millions of dollars in damages not just because they pushed a bad update, but because they pushed an update that ignored their customers' staging policies which would have prevented this type of widespread damage. Unbelievable.
Link to video:
Re: CrowdStrike to Delta: Stop pointing at us
#19> a “misleading narrative” that the cybersecurity company was responsible for the airline’s tech decisions and response to the outage Am I reading this right that this amounts to "you decided yourself to install our software on your devices, you should have known better?"
> “Should Delta pursue this path, Delta will have to explain to the public, its shareholders, and ultimately a jury why CrowdStrike took responsibility for its actions—swiftly, transparently, and constructively—while Delta did not,” wrote Michael Carlinsky, an attorney at law firm Quinn Emanuel Urquhart & Sullivan.
or maybe it's just how Delta didn't take their offer for help
> The cybersecurity company reiterated its apology to Delta for the initial disruption and said it had offered on-site assistance to Delta but was told it wasn’t needed. It said Bastian didn’t respond to outreach from CrowdStrike’s CEO.
Re: CrowdStrike to Delta: Stop pointing at us
#20> CrowdStrike said Sunday that its liability is contractually capped at an amount in the “single-digit millions.” Companies handling critical infrastructure should face more scrutiny imo.
Crowdstrike is not handling critical infrastructure. Delta is. The reality is the industry wants its cake and eat it too. No one forced Delta to buy a software which could force upgrades in their production fleet. They're a billion dollars company, and should put their big boys pants on.
lol.