Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Wells Fargo completely offline

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 256 posts

Re: Tell HN: Wells Fargo completely offline

#141

Post by habosa »

For those who have worked in bank IT, I have some questions.

So these days there's very little physical money. Most of my "wealth" is just entries in various digital ledgers. My bank says I have $XXX and my brokerage says I have $YYY and my retirement account says I have $ZZZ.

Let's go with the bank account case. Is it possible that a catastrophic accident or attack could wipe my balance down to $0 with no way to recover? What if a data center was nuked? What if two data centers were nuked?

How much redundancy is in the system? Are there third-party agencies that track private bank ledgers? How hard is it to take them out too?

Ever since I read "The End of Alchemy" (a great book, btw) this thought has haunted me.

Re: Tell HN: Wells Fargo completely offline

#142

Post by jstarfish »

paul7986 wrote: post #121
mabbo wrote: post #12

Earlier quoted context omitted.

Incredible. They had all their mission-critical infrastructure in a single data center. How many billions of dollars do they make per year? And they can't afford even a tiny bit of redundancy? If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.

What all their scandals didn’t convince you of that? After all their logo is that of a stage coach aka the wild Wild West(robbers, thieves, etc). They do not hide who they really are.

The stagecoach refers to their origins as an armored car/delivery service.

Re: Tell HN: Wells Fargo completely offline

#143

Post by kamikaz1k »

surge wrote: post #133
vonmoltke wrote: post #128

Earlier quoted context omitted.

Bloomberg does it once a year. That said, I have yet to encounter a company more obsessed with business continuity. I don't doubt that their failover systems and testing of them are well beyond the typical.

I think Bloomberg can stand to be down for 8 hours to simulate a disaster. Banks with legacy systems and people constantly dependent on them to conduct business can't risk an actual incident happening because they were testing what would happen if an incident happened. Netflix designed their stuff from the ground up to fail over. Large monolith corporations who've inherited systems from other companies they've bought…

Well that's not true.

Chaos engineering and AWS weren't real things when they started building the company. And the system they have now doesn't resemble much of it was once.

Truth of the matter is they invested more in their infrastructure, but that's because their business plan required them to grow on the back of technological advances. Banks, it's seems, do not. Or maybe they do, and the some of these start up banks will usurp them.

Re: Tell HN: Wells Fargo completely offline

#144

Post by Zelphyr »

nostrademons wrote: post #93
mywittyname wrote: post #68

Earlier quoted context omitted.

Lots of old companies are like this. They don't invest in critical infrastructure because it would bring them below their 20% margins.

Wells is weird though. When I started banking with them in 2009, they were ahead of the curve in online infrastructure - it was much easier to access them online than any of the banks I was used to back in Massachusetts. They also had a reputation for being both honest and conservative - they were the only AAA-rated bank in 2007, they were #1 in green rankings, and Warren Buffett had invested in them because of their…

In 2009/2010 my wife accidentally made a charge to PayPal that was linked with our Wells Fargo checking account. Since we weren’t planning on that charge happening from that account, there was an NSF fee and Wells rejected the charger. Understandable, right?

Except then PayPal made a few more attempts for god knows why and each time Wells Fargo kicked an NSF fee our way.

Now, PayPal shouldn’t have repeatedly attempted a rejected charge. But, Wells Fargo shouldn’t have allowed those attempts. They just couldn’t help themselves to that $35 NSF fee though.

We fought it to no avail. With all the NSF fees and interest (and fees they added to fees while we fought it), what started as a $300 transaction ultimately cost us over $1200.

Wells Fargo is now and was in 2009/2010 a criminal enterprise.

Re: Tell HN: Wells Fargo completely offline

#145

Post by ex3ndr »

gniv wrote: post #110
benj111 wrote: post #100

Earlier quoted context omitted.

What kind of redundancy are we talking about here? You can't really roll back say 10 minutes of transactions, so are you maintaining 2 parallel systems? How do you keep them perfectly in sync? This isn't my area of expertise by a long shot, but it occurs to me this is probably hard, especially when your codebase started in the 60s, and has been accreting ever since.

Yes it’s hard but there are mature solutions. See Google Spanner for example.

Spanner is slow as hell

Re: Tell HN: Wells Fargo completely offline

#146

Post by deevolution »

perl4ever wrote: post #136
deevolution wrote: post #35

Earlier quoted context omitted.

Queue infinite stream of downvotes. News flash, you dont need an exchange to store or transact bitcoin!!! And people using exchanges should understand the risks associated with using a centralized service.

What do you mean by "need"? If you don't have an exchange you can trust, doesn't that make things worse, because you have to determine whether you can trust your counterparties and you have to secure your assets with your own resources? Matt Levine wrote the other day: "...people think they don’t trust the traditional financial system, but in fact its most basic functionality—not forgetting where it put your money—is…

The main argument here is about the resiliency of the bitcoin network. Whatever your talking about has nothing to do with it.

Re: Tell HN: Wells Fargo completely offline

#147

Post by rolph »

dawnerd wrote: post #111
driverdan wrote: post #97

Earlier quoted context omitted.

I signed up for Stripe in 2011 or 2012. About six months after I signed up an account rep from Wells Fargo called me asking about my merchant account. Stripe never made it quite clear what was going on but apparently in the early days they opened accounts at Wells Fargo on your behalf without telling you.

Wasn’t Wells Fargo caught for making accounts for people without permission just to boost their numbers?

Something like this?

www.bloomberg.com/opinion/articles/2016-09-09/wells-fargo-opened-a-couple-million-fake-accounts

Re: Tell HN: Wells Fargo completely offline

#148

Post by deevolution »

perl4ever wrote: post #140
shawnz wrote: post #76

Earlier quoted context omitted.

> You need bitcoin exchanges for any kind of practical transaction. Not only that, people who don't keep their money on exchanges lose their coins all the time. OK, what's your point? Nobody claimed it was perfect, just that it happens to be immune to the particular failure mode which happened here. Can you accept that that is true? > This stuff just keeps on happening, but crypto folks always make some excuse as to…

"it happens to be immune to the particular failure mode which happened here" What exactly do you mean by "immune to the particular failure mode"? Is bitcoin immune to server or data center outages? Perhaps if you very narrowly limit which ones you are talking about to the bitcoin network per se.

Its not immune, no system is absolutely immune to failure, but it is highly resillient because of the wide distribution of its nodes and the fact that each node carries a full copy of the ledger.

Re: Tell HN: Wells Fargo completely offline

#149

Post by rolph »

OK, Hanlon's razor applies here:

https://en.wikipedia.org/wiki/Hanlon%27s_razor

recall Stuxnet, its possible this was an attack, somesort of malicious mod to firmware, but_ Hanlon's razor.

from the reddit:

"throwawayfordays75 1399 points 8 hours ago*2

Throwaway since I have first hand knowledge. Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the network or compute equipment and some failovers did not work as expected. "

At this point im wondering what "utility work" was happening.

Re: Tell HN: Wells Fargo completely offline

#150

Post by themodelplumber »

tatersolid wrote: post #101
legohead wrote: post #94

Earlier quoted context omitted.

I imagine a credit union would be susceptible to the same type of thing (single location/point of failure) due to their smaller sizes.

Credit unions and small banks outsource almost all of their core systems to companies like Fiserv. Literally thousands of banks and CUs all ride their infrastructure

I ran a website for a local CU for about 10 years. It was pretty fun doing things for their website that made their Fiserv web tools look like they were from 1970.

There was no interaction with user or account data on my end, just CMS-building, letting them go crazy with coupon promotions or pet shelter PSA's or warnings about ATMs being down, etc. They'd pitch me on a new project, I'd send them a timeline, and we'd add new features or tools to the website. I even hired a local college student to create some super-basic but useful financial tools for the site.

I loved how small-town-feel the whole thing was. At the beginning of the relationship, they gave me a list of broad requirements like "SAS-70", I found a DC to match, I sent them a contract, and I couldn't believe I actually had a banking institution as a client, 4 years out of school and a brand-new business owner.

Eventually they merged with another CU and went away, but apart from the very occasional "server down" notifications while I was on vacation, those are some really fun memories.

Post reply on HN