Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Overheating datacenter stopped 2.5M bank transactions

theregister.com

Post reply on HN
Print view View original

61–70 of 91 posts

Re: Overheating datacenter stopped 2.5M bank transactions

#61

Post by hinkley »

dylan604 wrote: post #7
throw0101c wrote: post #3

Seems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues." Perhaps regularly fl…

"No recovery plan is worth anything if it is not tested regularly." "An untested recovery plan is worth less than the paper on which it is printed." I'm sure there are many other versions of the quote

Have you seen this video of the guy who on some sort of “adventure” outing? They are way up in the air on some sort of platform where you are supposed to jump across these planks from one platform to another. Someone is filming from either a platform or plateau next to the rig.

There’s a harness connected to a rail above you to catch you if you fall.

In the video the man gets halfway across and the cable attached to his harness just pops off. The instructor behind him instinctively reaches out as if he can use The Force to catch him if he biffs, but he makes it to the other side unharmed, and is all smiles until the cable catches up and bops him in the back. At which point he turns around and connects the dots.

It’s a bit like that. And this is the image that comes into my mind every time I discover we have untested recovery processes while we are in the middle of a recovery.

Re: Overheating datacenter stopped 2.5M bank transactions

#62

Post by devaiops9001 »

We are no longer in an era where this should be allowed to happen, even for say... a legacy application that runs on Windows 2000 and needs NTFS on a network block device.

After 9/11, the "zero nines" whitepaper was released to push for cloud (never let a disaster go to waste).

I have hands-on deployed many K8s+Ceph clusters. Ceph's block storage can be deployed multi-AZ where multiple datacenters each with their own power systems are in close proximity. There are methods like Raft (used by K8s and Patroni) and a really great method demonstrated by Google's CloudSQL that uses a SQL table to infer which SQL server should be a "leader".

So like wtf guys.

Re: Overheating datacenter stopped 2.5M bank transactions

#63

Post by hinkley »

kevincox wrote: post #60
hinkley wrote: post #57

Earlier quoted context omitted.

Sticking your face in front of a pit bull you’ve never met before is “a strategy”. If you’re going to use the word strategy this way it has no meaning, which kills the conversation and communication in general. I think you’re confusing strategy and behavior. Strategy is a plan to do better than default behavior.

Sticking your face in front of a pit bull is a strategy for what? Writing up a plan for disaster recovery is a strategy for disaster recovery. The pros are that it is cheap and ticks the regulatory checkboxes. The downside is that it has a high probability of a high recovery time in the case of a disaster. My point is that there are different pros and cons to every strategy that you can pick. I also don't think that…

> Sticking your face in front of a pit bull is a strategy for what?

For meeting dogs of course.

The problem I have is that it seems like you agree with other people in this thread, but real world experience tells me that stating things this way is more fuel for the lazy and for anarchists than it is support for the SREs.

Don’t let the idiots think for a moment they are entitled to use a word as heavy as strategy. It ends farcically.

Writing it down doesn’t make it a strategy. Satire is also written down.

Re: Overheating datacenter stopped 2.5M bank transactions

#64

Post by bobbiechen »

cm2187 wrote: post #31

How to make this problem even worse: make all the banks in the street use the same datacenters, namely either AWS or Azure.

In the EU, the Digital Operational Resilience Act (DORA) mandates that financial services companies must manage their cloud risk - https://www.digital-operational-resilience-act.com/

Could lead to more multi-cloud adoption among financial services companies who are in scope.

(no relation to the other DORA, DevOps Research and Assessment, by the way)

Re: Overheating datacenter stopped 2.5M bank transactions

#65

Post by someplaceguy »

CyberDildonics wrote: post #58
someplaceguy wrote: post #42

Earlier quoted context omitted.

> You lost some of my message. So what's your point? Your point is that the balance will be reflected in the first layer after some time instead of being reflected after each and every lightning transaction? What does that have to do with what we were talking about? The point of this thread was how many transactions per time period you could do with Bitcoin, not whether the balance will be public information in real…

Lightning allows you to have an effectively limitless transaction rate Transactions that don't count and aren't useful to anyone until they become a bitcoin transaction, and that's still expensive and slow. No one cares about it because it is terrible technology that doesn't actually solve anyone's problems, because the people that made it made the problems in the first place.

> Transactions that don't count and aren't useful to anyone until they become a bitcoin transaction,

I have no idea why you're making such an obviously false assertion.

These transactions obviously "count" and are useful to the entities running those 15K Lightning nodes (many on behalf of multiple customers), which can all transact amongst themselves over the network, even without touching the Bitcoin network (or occasionally doing it just to settle balances).

These transactions have almost exactly the same cryptographic and trustless assurances as normal bitcoin transactions, which are far superior assurances than those provided by traditional payment / money transfer technologies.

> No one cares about it because it is terrible technology that doesn't actually solve anyone's problems

I think you are confusing your own ideas and problems with other people's ideas and problems.

Just because you don't care about it doesn't mean that other people don't (or that even you won't in the future, if your circumstances change).

Re: Overheating datacenter stopped 2.5M bank transactions

#66

Post by oooyay »

kevincox wrote: post #60
hinkley wrote: post #57

Earlier quoted context omitted.

Sticking your face in front of a pit bull you’ve never met before is “a strategy”. If you’re going to use the word strategy this way it has no meaning, which kills the conversation and communication in general. I think you’re confusing strategy and behavior. Strategy is a plan to do better than default behavior.

Sticking your face in front of a pit bull is a strategy for what? Writing up a plan for disaster recovery is a strategy for disaster recovery. The pros are that it is cheap and ticks the regulatory checkboxes. The downside is that it has a high probability of a high recovery time in the case of a disaster. My point is that there are different pros and cons to every strategy that you can pick. I also don't think that…

I think I can help disambiguate a bit. "is not a strategy" is a bit of a tip of the hat to "hope is not a strategy", the full extrapolation of which means that a plan is not equivalent to strategy and an unexercised strategy is the definition of hope.

An analog is when I was in the military we wore every level of MOP gear once a year to ensure that your gear functioned correctly and that you knew how to wear it. On a second day we would enter building filled with chemical irritant where you must dawn your mask, clear it, breath, and communicate. All of this is part of proving the plan which equivocates to strategy. We've not faced an imminent CBRN threat in decades, but we still do it because we maintain a strategy for dealing with it.

In this case, if you have a plan for how to deal with too much heat or too little power but you don't actively put the systems that transfer the responsibility or get put under stress during these disaster events then you never actually know for certain that they'll work. Therefore, they are not strategies, just plans.

Re: Overheating datacenter stopped 2.5M bank transactions

#67

Post by ms_frizzle »

jimt1234 wrote: post #55
psychlops wrote: post #29

The overheating datacenter didn't stop the transactions. It was the technical inability of the banks to fail over which should have been seamless. Outages happen.

IMHO, "fail over" will always fail. Any multi-data-center architecture should actively utilize all data centers, all the time, maybe dedicate more traffic to one or the other. But, if you've got a "passive" data center that you wanna activate during an outage, you're gonna have a bad day. Every time.

My experience contradicts your hyperbole. Over the past two decades I've been part of several fail overs with active/passive and build from scratch DR solutions. Most succeeded. The ones that didn't succeed on the first attempt did so on the second one. Not a perfect track record, but it does happen with proper documentation, testing, and skilled staff to handle it.

Re: Overheating datacenter stopped 2.5M bank transactions

#68

Post by stuff4ben »

HideousKojima wrote: post #12
throw0101c wrote: post #3

Seems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues." Perhaps regularly fl…

Or you could do chaos engineering: https://en.m.wikipedia.org/wiki/Chaos_engineering

I would venture to say that most non-tech startup companies are not equipped with the knowledge, the hardware, or the political fortitude to do something like that. Especially when it's customer's money at stake. It's one thing if Joe Schmo can't watch the latest Great British Baking Show. It's another matter entirely if Joe's paycheck is missing.

Re: Overheating datacenter stopped 2.5M bank transactions

#69

Post by someplaceguy »

eli wrote: post #59
chrischattin wrote: post #41

Earlier quoted context omitted.

Can't tell if sarcastic or not. But, that's perfectly legitimate. You could do that and if everything is signed correctly, the network will accept it. I could write down a seed phrase on a piece of paper and hand it to you, and that's a completely offline transaction.

You don't need a network at all. Just email me who you want to transact with and I'll create a row for them.

Ok, so can I now redeem the BTC that you sent me earlier please, so that I can spend them in the first layer (or the second one)? Given that nobody else seems to want to use your third layer.

The fact that I even have to ask you already demonstrates how shitty your third layer is compared to Lightning, not to mention all the other obvious ways in which it's infinitely worse.

Re: Overheating datacenter stopped 2.5M bank transactions

#70

Post by paulddraper »

BirAdam wrote: post #19
paulddraper wrote: post #11

Earlier quoted context omitted.

Probably observer's bias, but I've never seen a recovery plan work like it's supposed to.

I’ve worked a few places where the company did regular disaster recovery testing to ensure stuff worked how it was intended to. Most of the time, it worked. Every once in a while, it failed.

Yeah the "testing" worked....but did it really?

Maybe I'm just too skeptical.

Post reply on HN