Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Wells Fargo completely offline

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 256 posts

Re: Tell HN: Wells Fargo completely offline

#151

Post by vonmoltke »

surge wrote: post #133
vonmoltke wrote: post #128

Earlier quoted context omitted.

Bloomberg does it once a year. That said, I have yet to encounter a company more obsessed with business continuity. I don't doubt that their failover systems and testing of them are well beyond the typical.

I think Bloomberg can stand to be down for 8 hours to simulate a disaster. Banks with legacy systems and people constantly dependent on them to conduct business can't risk an actual incident happening because they were testing what would happen if an incident happened. Netflix designed their stuff from the ground up to fail over. Large monolith corporations who've inherited systems from other companies they've bought…

> I think Bloomberg can stand to be down for 8 hours to simulate a disaster. Banks with legacy systems and people constantly dependent on them to conduct business can't risk an actual incident happening because they were testing what would happen if an incident happened.

No, it can't. Any loss of customer-facing functionality is a critical outage ("World Problem" in company terminology). There are a relatively small number of customers, but the terminal is critical to the operations of those who buy it. The terminal going down for eight hours would be a world-wide headline in the financial press.

A Tier 1 test that simulates loss of a datacenter takes a cluster one DC virtually offline. This puts an entire subset of services offline in that DC entirely. The test is coordinated with the teams who own the services to ensure their services fail over correctly. Any service disruption during the failover is a test failure. If it passes, the customers don't even know it happened. The goal is to be able to lose an entire DC and have the terminal customers not realize it until they hear about it on the news.

Re: Tell HN: Wells Fargo completely offline

#152

Post by ianai »

kamikaz1k wrote: post #143
surge wrote: post #133

Earlier quoted context omitted.

I think Bloomberg can stand to be down for 8 hours to simulate a disaster. Banks with legacy systems and people constantly dependent on them to conduct business can't risk an actual incident happening because they were testing what would happen if an incident happened. Netflix designed their stuff from the ground up to fail over. Large monolith corporations who've inherited systems from other companies they've bought…

Well that's not true. Chaos engineering and AWS weren't real things when they started building the company. And the system they have now doesn't resemble much of it was once. Truth of the matter is they invested more in their infrastructure, but that's because their business plan required them to grow on the back of technological advances. Banks, it's seems, do not. Or maybe they do, and the some of these start up ba…

Standard good practice should be to have a redundancy in place and test it at a regular interval. It should be part of periodic maintenance - fail to the backup so updates/grades can be applied to production and fail back to production once done.

But I’m guessing wellsfargo just doesn’t have a reason to care.

Re: Tell HN: Wells Fargo completely offline

#153

Post by 40acres »

habosa wrote: post #141

For those who have worked in bank IT, I have some questions. So these days there's very little physical money. Most of my "wealth" is just entries in various digital ledgers. My bank says I have $XXX and my brokerage says I have $YYY and my retirement account says I have $ZZZ. Let's go with the bank account case. Is it possible that a catastrophic accident or attack could wipe my balance down to $0 with no way to rec…

Generally I think here is where the FDIC and SPIC would come in.

Re: Tell HN: Wells Fargo completely offline

#154

Post by hinkley »

edoceo wrote: post #63
CydeWeys wrote: post #55

Earlier quoted context omitted.

Unfortunately they're the holders of my mortgage (it was sold twice and they ended up with it). So I don't even have a choice of being their customer for the next ~25 years.

Refi that bad-boy to your local credit union

It doesn't work that way. Mortgages get resold between banks all the time. You can walk into your local bank and get a house loan, and five months from now you'll get a business envelope from Wells Fargo informing you that you should send your checks to them now.

I would expect if WF decided your first loan met their purchasing criteria, your refinance would get the same treatment.

Re: Tell HN: Wells Fargo completely offline

#155

Post by zepearl »

dpcan wrote: post #137
mabbo wrote: post #12

Earlier quoted context omitted.

Incredible. They had all their mission-critical infrastructure in a single data center. How many billions of dollars do they make per year? And they can't afford even a tiny bit of redundancy? If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.

Okay, so say you switch banks. How do you have ANY idea if your new bank is better or worse? Are you going to ask customer support? They barely know how to access your account let alone the technical layout of their data centers.

I know that both Switzerland and Singapore have at least "soft" guidelines (no direct retaliation if you don't adhere to them, but frown upon if you don't) about the HW (including buildings & workplaces) & SW and personnel setup required to remediate a potential catastrophic failure of the data centers and/or key-employee's office building.

Example of high-level guidelines (Singapore): http://www.mas.gov.sg/~/media/resource/legislation_guideline...

I think that in Switzerland all major banks test their disaster-readiness (by switching everything to their secondary datacenters & working locations) of all critical applications/software-layers and employees at least once every 3 years - reaction/recovery times depend on the criticality of the service provided by the person/application.

Re: Tell HN: Wells Fargo completely offline

#156

Post by mbesto »

nodesocket wrote: post #34

I thought at one time Stripe used Wells Fargo as their bank. Any reported problems from Stripe?

"Wells Fargo as their bank" is a weird statement. Wells Fargo is a HUGE company with various banking services. This isn't as simple as "WF data center had an outage so WF as a company is completely down"...that's not how it works.

Re: Tell HN: Wells Fargo completely offline

#157

Post by icelancer »

surge wrote: post #133
vonmoltke wrote: post #128

Earlier quoted context omitted.

Bloomberg does it once a year. That said, I have yet to encounter a company more obsessed with business continuity. I don't doubt that their failover systems and testing of them are well beyond the typical.

I think Bloomberg can stand to be down for 8 hours to simulate a disaster. Banks with legacy systems and people constantly dependent on them to conduct business can't risk an actual incident happening because they were testing what would happen if an incident happened. Netflix designed their stuff from the ground up to fail over. Large monolith corporations who've inherited systems from other companies they've bought…

> I think Bloomberg can stand to be down for 8 hours to simulate a disaster.

Do you know what Bloomberg does? It powers equities trading markets around the world, 24/7. It isn't just news.

Re: Tell HN: Wells Fargo completely offline

#158

Post by dbancajas »

clairity wrote: post #132
baby wrote: post #127

Earlier quoted context omitted.

Is there a bank in the US similar to Monzo or Revolut? I really miss these having moved from the UK to the US and I haven't been able to find a good replacement.

i don't know enough about those banks to compare, but https://empower.me is the best of the new breed online banks i've found - 2% apy for checking (with some reasonable conditions)!

how long have you had this?

Re: Tell HN: Wells Fargo completely offline

#159

Post by pushtheenvelope »

surge wrote: post #115
mabbo wrote: post #12

Earlier quoted context omitted.

Incredible. They had all their mission-critical infrastructure in a single data center. How many billions of dollars do they make per year? And they can't afford even a tiny bit of redundancy? If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.

They had redundancy, the redundancy failed. I mean truthfully when do you get to test your redundancy against a true disaster. It was a mess. WF is 20 companies rolled into one so the fact the disparate systems from 10 different banks works at all is kind of a miracle.

Facebook regularly takes down multiple data-centers at a time to stress test this. Its users rarely notice (which I think is the point).

Re: Tell HN: Wells Fargo completely offline

#160

Post by bowmessage »

40acres wrote: post #153
habosa wrote: post #141

For those who have worked in bank IT, I have some questions. So these days there's very little physical money. Most of my "wealth" is just entries in various digital ledgers. My bank says I have $XXX and my brokerage says I have $YYY and my retirement account says I have $ZZZ. Let's go with the bank account case. Is it possible that a catastrophic accident or attack could wipe my balance down to $0 with no way to rec…

Generally I think here is where the FDIC and SPIC would come in.

Do they also have backups of account balances across all insured accounts then?
Post reply on HN