Live data from Hacker News

Why the CrowdStrike bug hit banks hard

bitsaboutmoney.com

211–220 of 250 posts

Re: Why the CrowdStrike bug hit banks hard

#211
post #91

Did it really hit banks hard? Core banking systems don't run windows, they run on mainframes typically on IBM z/OS. I know it hit the financial firms hard and knocked out their trading systems but I don't know of any major bank losing their core bank system due to crowdstrike. Australia got hit hard because they modernized their bank systems and now most are cloud based. I am not aware of any major bank running their…

[dead]

Re: Why the CrowdStrike bug hit banks hard

#212
post #97

The article specifically mentions US banks and as I personally didn't see any disruption over here - is there (anec)data on how popular CrowdStrike is in the US vs the EU?

Can't have disruption from CrowdStrike if you run on IBM mainframes with cobol coz your math only opens gates for new technologies once in 25 years.

Oh wow an Anathem reference!

To answer the question, CrowdStrike is a global company with thousands of employees around the world. Not sure why the EU wasn't hit as hard.

Re: Why the CrowdStrike bug hit banks hard

#213
> Configuration bugs are a disturbingly large portion of engineering decisions which cause outages

I work in medical device software -- the stuff that runs on machines in hospital labs, ER's or at patient bedside.

The first "ohmigod do we need to recall this?" bug I remember was an innocuous piece of code that was inserted to debug a specific problem, but which was supposed to be disabled in the "non-debug" configuration.

Then somehow, the software update shipped with a change to the configuration file that enabled that code to run. Timing-critical debug code running on a real-time system with a hard deadline is a recipe for disaster.

Thankfully, we got out of that pretty easily before it affected more than a small handful of users, but things could have been a lot worse.

Re: Why the CrowdStrike bug hit banks hard

#214

Earlier quoted context omitted.

Microsoft was on their way to doing this, but was shot down by EU regulators because the APIs weren't available to all third-party vendors.

I think it's kind of ridiculous to then blame the regulators for the fact that Microsoft decided not to go ahead with a more competitor-friendly design. The fact that Microsoft abandoned it as soon as a regulator pointed out how anti-competitive the design of the API was makes you wonder what Microsoft's true intention was. To me that implies the anti-competitive design was its main feature and to Microsoft it would'…

Maybe. Not working at MS I can't say what their reasons were.

But another way of looking at this would be that perhaps they wanted to be the beta testers of the API themselves because opening it up would have been a maintenance liability for the company. Microsoft tends to be pretty good about backwards compatibility in ways that Apple is not.

We also don't know that these APIs were cancelled, they may make it into future versions of windows.

Re: Why the CrowdStrike bug hit banks hard

#215
post #209

Earlier quoted context omitted.

Yes, you are arguing for that microwave when you argue Microsoft should approve the software you're allowed to run on a Windows box and be liable for its performance. Should Microsoft also have to approve what browsers you're allowed to run, should they approve what chat applications you're allowed to use? And sure, why shouldn't you be able to modify the software on hardware you own? It's your microwave. If you modi…

I'm not making an analogy with the microwave (your saying food is software and the microwave is hardware) I'm literally talking about the software that runs on a microwave.

I'm aware of the point you're trying to make with the microwave. I'm making another analogy; one you're not getting. And either way, yes, I think you should be able to change the software on the microwave. It is your microwave. Do whatever you want with it. Why should Samsung or GE have the right to say what you can or cannot do with the things you own?

If we want to talk microwaves, Microsoft is the microwave manufacturer. Users installing CrowdStrike are people sticking a giant ball of foil and paper towels in the microwave and turning it on for an hour. You're arguing Microsoft is liable for the things people stick in their microwaves, and that Microsoft should put in place guards to prevent people from putting whatever they want in their own microwaves. That Microsoft should control the things people put in their microwaves. Only Microsoft tested and Microsoft approved foods in Microsoft microwaves. And the microwave needs to ensure only the proper cook time applies to the properly signed food products to make sure it doesn't get burnt. Sorry, Microsoft hasn't fully validated Red Gold potatoes, it can only cook Russet potatoes.

That is the same logic as Microsoft is liable for the third-party software people install on Windows machines and that Microsoft shouldn't have allowed the third-party software to run.

Why should Microsoft be able to say what antivirus software I choose to install or not? Why should Microsoft be able to say what browser I install? If I install some software that breaks my Windows machine, is that the faut of Microsoft or the fault of the software maker? If I stick foil in the microwave is the ensuing fire GE's fault?

Re: Why the CrowdStrike bug hit banks hard

#216
post #201

Earlier quoted context omitted.

The update bypassed the controls orgs had in place to defer/schedule updates, AFAIK.

Didn't day say in their incident report that they have a batched rollout strategy for software updates but this was a config update and the update path for configs does not have such a mechanism in place.

Ya, so hopefully it's obvious to them that every rollout needs some kind of batching. I get that all devices within one org might need to have the same config, but in that case batch it out to different orgs over 2-3 days.

Maybe the more critical infrastructure and health care orgs are at the end of that rollout plan so they are at lower risk. It's not ideal if one sandwich shop in Idaho can't run their reports that day, but that's far better than shutting down the hospital next door. CrowdStrike could even compensate those one system shops that are on the front line when something goes down.

Again, better to pay a sandwich shop a few thousand dollars for their lost day of sales than get sued by the people in the hospital who couldn't get their meds, x-rays, etc in time.

Re: Why the CrowdStrike bug hit banks hard

#217

While reading this I was struck with an interesting question: What risk does any particular software vendor pose to an industry at large? For example (making up numbers here): if 75% of all airline computers have croudstrike falcon installed that seems like a very concentrated risk. I actually wouldn't be surprised if we had this we would see really high concentrations of a small number of vendors in any industry.

Alternatively, if Oracle hikes the price on an industry-specific product by 75%, how much of that industry goes under?

Re: Why the CrowdStrike bug hit banks hard

#218
post #77

Earlier quoted context omitted.

The update bypassed the controls orgs had in place to defer/schedule updates, AFAIK.

I've had trouble nailing down if thats the case from searching around online. And if thats true - thats absolutely on Crowdstrike. And that behavior should disqualify it from being used on critical systems. I imagine this incident will cause a lot of teams to consider just what can happen automatically on their systems.

It's on CrowdStrike, but it's also on IT for even allowing installation of critical software like this that has a bypass at all. Updates shouldn't even be allowed to bypass IT's safe rollout procedures, at least not without IT signing off on it anyway.

Re: Why the CrowdStrike bug hit banks hard

#219
post #142

Earlier quoted context omitted.

So why didn't MS lock it down in the US if it's an EU-local rule? Their excuse isn't plausible.

You're spilling cheap propaganda. Microsoft likely never had[0] an appropriate userland-level API in place and them blaming the EU should not be repeated by someone calling themselves a journalist. [0] https://www.youtube.com/watch?v=EGttFWntctU - I need to state here that I do not possess the level of knowledge the author of video presents and therefore am unable to confirm findings included in the video

> Microsoft likely never had

And we're back to Microsoft -- they are responsible for not having a proper way to handle such third-party apps, nor they maintained a process and controls to prevent such rogue breaking updates.

Re: Why the CrowdStrike bug hit banks hard

#220
post #21
post #3

I'm still amazed how the blame shifted from Microsoft to CrowdStrike. Yes, CrowdStrike update caused that -- but applications fail all the time. It was Microsoft's oversight to put it on Windows critical path. And banks/airlines etc were hit hard because their _Windows_ didn't boot, not because of an application crash on a perfectly working Windows.

What about the previous crowdstrike bugs that hit Linux systems in a similar fashion? I don't understand how this has anything to do with Windows, Crowdstrike is the one who built the application.

It has everything to do with Windows, because it's Windows who crashed.

Applications crash all the time. But in this case people weren't able to even load the Windows to figure what's wrong or what app has crashed.

Microsoft allowed a third-party to self-update and didn't put a proper system of review and updates control to the heart of its OS.

Post reply on HN