Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Overheating datacenter stopped 2.5M bank transactions

theregister.com

Post reply on HN
Print view View original

11–20 of 91 posts

Re: Overheating datacenter stopped 2.5M bank transactions

#11

Post by paulddraper »

throw0101c wrote: post #3

Seems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues." Perhaps regularly fl…

Probably observer's bias, but I've never seen a recovery plan work like it's supposed to.

Re: Overheating datacenter stopped 2.5M bank transactions

#12

Post by HideousKojima »

throw0101c wrote: post #3

Seems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues." Perhaps regularly fl…

Or you could do chaos engineering: https://en.m.wikipedia.org/wiki/Chaos_engineering

Re: Overheating datacenter stopped 2.5M bank transactions

#13

Post by someplaceguy »

CyberDildonics wrote: post #10
someplaceguy wrote: post #5

Earlier quoted context omitted.

Two nodes in the Lightning network [1] could right now be performing 5M transactions per day and you'd be none the wiser (unless they'd tell you, that is). And that's just 2 nodes out of the current 15K-20K. [1] i.e. Bitcoin second layer.

Except that those transactions don't count until they get on the main chain and no one needs to send 1000 transactions of single penny, then have the exact same problems as everyone else when trying to get that to actually be usable.

> those transactions don't count

They don't? Then surely you wouldn't mind sending me a few bitcoins over a lightning channel? Pretty please?

> no one needs to send 1000 transactions of single penny

There is no such "single penny" limit.

> then have the exact same problems as everyone else

What problems?

Re: Overheating datacenter stopped 2.5M bank transactions

#14

Post by stuff4ben »

throw0101c wrote: post #3

Seems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues." Perhaps regularly fl…

We used to do this for some large Artifactory clusters (~1bn artifacts) we ran on both the east and west coast of the US. Failing over to our DR cluster and our (internal) global DNS load balancer handled it just fine. Made upgrading those clusters seamless for the most part. Gave me confidence we could continue development in the event our main site fell into the ocean for some reason.

Re: Overheating datacenter stopped 2.5M bank transactions

#15

Post by SandraBucky »

throw0101c wrote: post #3

Seems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues." Perhaps regularly fl…

Bureaucracy triumphs over engineering considerations most of the time. Engineering division at banks might have suggested testing backup but top execs might disagree as it is unnecessary risk on operational system. Of-course they will play the blame game afterward and some low-level contractor will receive the heat cz of inability of decision making at top. Disclaimer: I worked as ELV contractor for an asset management megacorp.

Re: Overheating datacenter stopped 2.5M bank transactions

#17

Post by u32480932048 »

CyberDildonics wrote: post #10
someplaceguy wrote: post #5

Earlier quoted context omitted.

Two nodes in the Lightning network [1] could right now be performing 5M transactions per day and you'd be none the wiser (unless they'd tell you, that is). And that's just 2 nodes out of the current 15K-20K. [1] i.e. Bitcoin second layer.

Except that those transactions don't count until they get on the main chain and no one needs to send 1000 transactions of single penny, then have the exact same problems as everyone else when trying to get that to actually be usable.

Worse, Lightning won't let you send pennies, so you have to send 10,000,000 transactions of 0.0000000001 BTC - each transaction consuming the equivalent energy of a big lightning bolt the size of a small lightning bolt (hence the name).

Re: Overheating datacenter stopped 2.5M bank transactions

#18

Post by someplaceguy »

u32480932048 wrote: post #17
CyberDildonics wrote: post #10

Earlier quoted context omitted.

Except that those transactions don't count until they get on the main chain and no one needs to send 1000 transactions of single penny, then have the exact same problems as everyone else when trying to get that to actually be usable.

Worse, Lightning won't let you send pennies, so you have to send 10,000,000 transactions of 0.0000000001 BTC - each transaction consuming the equivalent energy of a big lightning bolt the size of a small lightning bolt (hence the name).

What the heck are you talking about?

None of what you just said makes any sense.

Re: Overheating datacenter stopped 2.5M bank transactions

#19

Post by BirAdam »

paulddraper wrote: post #11
throw0101c wrote: post #3

Seems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues." Perhaps regularly fl…

Probably observer's bias, but I've never seen a recovery plan work like it's supposed to.

I’ve worked a few places where the company did regular disaster recovery testing to ensure stuff worked how it was intended to. Most of the time, it worked. Every once in a while, it failed.

Re: Overheating datacenter stopped 2.5M bank transactions

#20

Post by pphysch »

dylan604 wrote: post #7
throw0101c wrote: post #3

Seems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues." Perhaps regularly fl…

"No recovery plan is worth anything if it is not tested regularly." "An untested recovery plan is worth less than the paper on which it is printed." I'm sure there are many other versions of the quote

But think of all the boxes you cannot check if you are honest about your recovery plan
Post reply on HN