Its not your fault it happened, but it is your responibility to deal with it.
edit: delta should have had better system outage processes
CrowdStrike to Delta: Stop pointing at us
71–76 of 76 posts
Re: CrowdStrike to Delta: Stop pointing at us
#72One issue that hasn't received enough attention comes from a comment on Dave Plummer's video on the CrowdStrike outage. Dave Plummer is a former Windows engineer and runs a YouTube channel call Dave's Garage. @zug-zug wrote: > While this is technically what crashed machines it isn't the worst part. > CS Falcon has a way to control the staging of updates across your environment. businesses who don't want to go out of…
I'm pretty sure this is why everything we got in the first 48 hours from CS was stressing that the issue was with a "channel file" (threat definitions, content updates, etc). Their staged update process is for the falcon driver itself. It is not for the "channel files". As I understand it, the driver itself is understood to be a risk, and they provide facility for an N, N-1, N-2 staged deployment to mitigate this ris…
That's an important distinction. CrowdStrike probably did, in fact, CYA in the licensing terms.
Re: CrowdStrike to Delta: Stop pointing at us
#73Earlier quoted context omitted.
> While I am sure that Delta's IT department was understaffed, this was also a unique situation. If you spent the time to make a well optimized machine for rolling out updates, things were automated, and you expected things to go wrong but I would have never anticipated every Windows machine being unable to boot. That is an extra-ordinary situation. I doubt any IT department is really staffed to be able to handle tha…
> Let this be a lesson to not introduce single points of failure into critical systems without having prepared for their unavailaility or misbehaviour. I think that is where this is going to get more complicated, I think that this has broadened what we traditionally think about when it comes to what we consider a single point of failure. I would wager that most people would not have considered something like Crowdstr…
> I would wager that most people would not have considered something like Crowdstrike to be a single point of failure. It is not what you would generally think of.
Sure. Most people are not decision-makers for critical IT infrastructure. Should we similarly throw our hands in the air if a bridge or building collapses because the people responsible didn't pay attention to structural safety just because it wasn't obvious to a layperson?
> Hell I would argue that in this particular situation, you could have the best disaster recovery but it wouldn't have done anything since Crowdstrike was probably baked into your images.
It's not really "the best disaster recovery" then, is it?
> Yeah eventually that would have been a non issue, but I can not imagine a scenario that I would have ever thought that I had a baked image and assuming I still had access to it and I could assure that it was the image that I made, that it somehow would itself be a problem.
There are so many other ways this can back-fire I'm not going to try to enumerate them. I really hope no one put you in charge of operating anything critical...
Re: CrowdStrike to Delta: Stop pointing at us
#74> CrowdStrike said Sunday that its liability is contractually capped at an amount in the “single-digit millions.” Companies handling critical infrastructure should face more scrutiny imo.
Crowdstrike is not handling critical infrastructure. Delta is. The reality is the industry wants its cake and eat it too. No one forced Delta to buy a software which could force upgrades in their production fleet. They're a billion dollars company, and should put their big boys pants on.
I mean. Delta is also an airline, and if airline's love to do one thing it's to point fingers and shift blame. Mostly such that they don't reimburse you what you are legally owed if they jam you up, but also it seems throughout.
Re: CrowdStrike to Delta: Stop pointing at us
#75Re: CrowdStrike to Delta: Stop pointing at us
#76Earlier quoted context omitted.
AFAIK crowdstrike can push updates at any time at any host. There are staging areas they may use, but don't have to (particularly for definitions updates). Crowdstrike should have done a better job , but Delta chose them (to offload the responsibility and work) and now they're claiming foul. They knew the risk. This is a classic executive play of claiming the fault lies in the consultants/vendor and taking no respons…
I'm not sure how "you should never use CrowdStrike" is an argument in CrowdStrike's favor. I guess you're saying they shouldn't have outsourced in the first place? Which does sound like the correct conclusion in this case...
I'm not trying to defend CrowdStrike, but pointing to the fact Delta is the one maintaining and owning critical infrastructure and the executives trying to shift this responsibility onto someone else is the reason this happened in the first place. :)