Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the CrowdStrike bug hit banks hard

bitsaboutmoney.com

Post reply on HN
Print view View original

241–250 of 250 posts

Re: Why the CrowdStrike bug hit banks hard

#241

Post by ctxc »

mewpmewp2 wrote: post #226
burnished wrote: post #188

Earlier quoted context omitted.

The difference is that if windows does the skipping then you probably don't find out until its too late, if the application does the skipping there is the opportunity to set up alerting so you can fix whatever went wrong.

Do you mean that the skip would be manually approved after telemetry is sent and folks on-call paged? Then that sounds like it could be viable and a good idea yes. But always a chance that the skipping mechanism could break as well. And there must be some form of networking available to able to send that and ask for approval.

Exactly! On skipping mechanism breaking - I mean, anything could break. Boils down to design and testing like all things.

One change - this approval and telemetry doesn't happen during the boot loading process. It's just logged and skipped.

Once bootup is done, the EDR app auto starts, checks logs for anomalies and sends telemetry over whenever network is available (it usually is, because they update malware signatures etc frequently). Someone at the company gets paged, they fix and the process continues.

Re: Why the CrowdStrike bug hit banks hard

#242

Post by pantulis »

SkyPuncher wrote: post #95
pantulis wrote: post #2

This is a good writeup, but to be fair it's just not a matter of banking regulations. Basically all big companies are under similar obligations regarding endpoint protection.

Basically all B2B companies are under some sort of obligation to have endpoint protection. All of these requirements essentially become transitive across a company's entire supply chain. * Big bank needs to comply with X, so do all of their vendors. * Vendor wants to sell to big bank, so they comply with X. They also need all of their vendors to comply with X. * So on and so on. ---- Ultimately, there are a lot more…

Yes, this is a good summary of the situation. As a matter of fact, I guess there were quite a lot of systems and services that went down even though they were not using Crowdstrike themselves, but some part of their cloud supply chain was. I see Salesforce and Adobe were impacted in some way, probably due to the collateral Azure disruption.

On the other hand, count me surprised at the sales prowess of Crowdstrike, I did not know how big they were.

Re: Why the CrowdStrike bug hit banks hard

#243

Post by ClumsyPilot »

deepsun wrote: post #223
ClumsyPilot wrote: post #73

Earlier quoted context omitted.

This is the comment I expected, begging to handover your freedoms to run software to a big carry. If you replace parts in your BMW, and put in some garbage or incompatible parts, it your fault if it doesn’t run. You expect to sue your mechanic if he messed up, and for him to cover the full cost. For some reason people do not expect CrowdStrike to pay for their stupidity, which is the root of the problem. And the mana…

Bit it wasn't some garbage parts in a car, it was an app. And apps fail all the time, OS is expected to handle that. Same as car is expected to handle rain for example.

> it was an app. And apps fail all the time

Exactly

The fact that developers do not take their responsibility as seriously as an average car mechanic bring shame on our entire industry

Re: Why the CrowdStrike bug hit banks hard

#244

Post by HeyLaughingBoy »

saltminer wrote: post #182

> This created a minor emergency for me, because it was an other-than-minor emergency for some contractors I was working with. > Many contractors are small businesses. Many small businesses are very thinly capitalized. Many employees of small businesses are extremely dependent on receiving compensation exactly on payday and not after it. And so, while many people in Chicago were basically unaffected on that Friday be…

The really admirable thing about this is that Patrick acknowledged that it was "an other-than-minor emergency" for the contractors and took steps to ensure that they were paid rapidly. In a similar situation many people would have shrugged and taken an attitude of "sorry, bank's down. I'll pay you when it comes back up."

Re: Why the CrowdStrike bug hit banks hard

#245

Post by btbuildem »

alephnerd wrote: post #47
tootie wrote: post #33

Earlier quoted context omitted.

This is going to crush their sales pipeline and lead to at least a few attempting a migration off. Crowdstrike is unlikely to go out of business, but this is not a good time to buy.

Safe Harbor: Don't follow random internet commentators opinions on public markets. This is just an opinion and not advice. I disagree. Long term, the fundamentals of CRWD continue to remain unabated. Endpoint protection is still a critical need no matter what - for every bug like CRWD, there's always a company you can point to who's operations were shut down due to an attack. CRWD skimped on QA and customer support,…

> They'll probably have a decently large layoff in the sales org

Bingo. That's the buy signal

Re: Why the CrowdStrike bug hit banks hard

#246

Post by __MatrixMan__ »

vel0city wrote: post #227
__MatrixMan__ wrote: post #65

I like the technical stuff here. I'm not so sure about this: > money is core societal infrastructure, like the power grid and transportation systems are. It would be really bad if hackers working for a foreign government could just turn off money. Sure, it would be inconvenient in the short term. But I think the current design is holding us back. I suspect that most of us would have more to gain than to lose if we ma…

The uber-wealthy don't have most of their assets in currency. Its in stocks, houses, cars, boats, etc. Delete the dollars, it'll hurt them a bit, but in the end they still have a house(es). But now all those people who were using currency to trade for housing now suddenly need to find a new way to trade for shelter. Who got hurt worse here?

I'm not going to try to lay down the exact parameters of what we'd come up with in money's place, but if it's going to be resilient in the face of far-away servers behaving badly then it would have to derive it's legitimacy not from some shiny golden ledger of who owns which dollar, house, or car, but instead from who is behaving in a way that benefits the people around them.

So yeah, it could go as you say, but only of the wealthy are behaving in a way that justifies their outsized share while the renters are just spending from a pile of money that they got through less honorable means.

I don't think that's the most likely scenario though.

Re: Why the CrowdStrike bug hit banks hard

#247

Post by deepsun »

School-Cotton wrote: post #8
deepsun wrote: post #3

I'm still amazed how the blame shifted from Microsoft to CrowdStrike. Yes, CrowdStrike update caused that -- but applications fail all the time. It was Microsoft's oversight to put it on Windows critical path. And banks/airlines etc were hit hard because their _Windows_ didn't boot, not because of an application crash on a perfectly working Windows.

Microsoft is not who made the decision to put this on Windows' critical path; CrowdStrike was. Nothing stops you from running whatever dodgy third-party kernel modules you like on Linux or FreeBSD and they could easily cause the same sort of problem.

Partially agree. Linux yes, but *BSD systems have microkernel architecture, so must be more resilient to failures of one of the components. Although I have no idea whether the full system would boot either, I'm pretty sure it could partially load, give more information to user, and make it easier to fix.

Re: Why the CrowdStrike bug hit banks hard

#248

Post by deepsun »

Bjartr wrote: post #23
School-Cotton wrote: post #8

Earlier quoted context omitted.

Microsoft is not who made the decision to put this on Windows' critical path; CrowdStrike was. Nothing stops you from running whatever dodgy third-party kernel modules you like on Linux or FreeBSD and they could easily cause the same sort of problem.

In fact, CrowdStrike has taken down Linux systems in much the same way in the past year (in April I think). It's just that the impact was less widespread.

Linux yes, but *BSD systems have microkernel architecture, so must be more resilient to failures of one of the components. Although I have no idea whether the full system would boot either, I'm pretty sure it could partially load, give more information to user, and make it easier to fix.

Re: Why the CrowdStrike bug hit banks hard

#249

Post by hello_moto »

lucianbr wrote: post #80
jmuguy wrote: post #58

Maybe the IT departments at the affected orgs take solace in the fact that so many other orgs had issues that the heat is off - but in my opinion this was still a failure of IT itself. There's no reason that update should have been pushed automatically to the entire fleet. If Crowdstrike's software doesn't give you a way to rollout updates on a portion of your network before the entire fleet, it shouldn't be used.

Management decides to use Crowdstrike, not IT, and IT has no way to rollout updates in controlled fashion. So not really a failure of IT, at least not for this reason.

In big companies, it's the Management of the IT team.

I know, not really the DailyWTF materials that majority HNers led to believe.

Re: Why the CrowdStrike bug hit banks hard

#250

Post by hello_moto »

lucianbr wrote: post #130
jmuguy wrote: post #92

Earlier quoted context omitted.

My comment assumes that the IT department (including its executive) gets to make these sort decisions - why wouldn't they?

Sure, if there's a security exec who made the decision, it may be their fault. I was thinking more of the rank and file, but that's just my bias.

When a ransomeware attack is happening, organizations will engage to cybersecurity vendors and will start PoC with a bunch of them and list the pros and cons of each vendors before they negotiate and ended up selecting the winner.
Post reply on HN