Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Wells Fargo completely offline

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 256 posts

Re: Tell HN: Wells Fargo completely offline

#181

Post by zhte415 »

mabbo wrote: post #12
nemothekid wrote: post #7

Apparently there was a fire in a Datacenter - theres a thread on /r/sysadmin by an insider. https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo...

Incredible. They had all their mission-critical infrastructure in a single data center. How many billions of dollars do they make per year? And they can't afford even a tiny bit of redundancy? If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.

I was shocked to learn the Fortune-500 bank I was working for had only a single data centre. Should they ever go down, much of the world would be disrupted. I think this is not unique.

Re: Tell HN: Wells Fargo completely offline

#183

Post by zdragnar »

turc1656 wrote: post #82
drharby wrote: post #74

Earlier quoted context omitted.

Real talk...they own my mortgage, i cant just 'forget about it' if they forget about it. I want to refinance it. Of they lose all data on my debt like that ending to Fight Club... is the debt just...gone?

Nope. Liens are recorded with the county/state as part of property law, as well as any other encumbrances. So that would actually probably make your situation worse because the lien holder would have no clue about the debt and claim on the property and could be hell for you to remove even if you kept paying and eventually paid everything off. Could potentially be a nightmarish scenario and an exercise in pure bureauc…

I'm relatively certain that it is on the lien holder to prove that you owe them money; otherwise, you could just re-file to have the lien removed.

Re: Tell HN: Wells Fargo completely offline

#184

Post by Aloha »

gdubs wrote: post #161
40acres wrote: post #153

Earlier quoted context omitted.

Generally I think here is where the FDIC and SPIC would come in.

I’d love someone who knows the details to explain how that claim process would work, because while the FDIC would insure in the event a bank goes under, in that scenario there’s usually records. What would happen if there were no records? Surely there’d be lots of people making significant deposits in between snapshots. If you’re “paperless” I’m not even sure how you’d reconstruct your balance. Would certainly be tim…

You presume the live system is the only record - its not, there are backups of those, then historical records to reconstruct what might have been there.

Bear in mind the live records are only one part of what a bank keeps.

Re: Tell HN: Wells Fargo completely offline

#185

Post by sdrothrock »

illumin8 wrote: post #96
WrtCdEvrydy wrote: post #13

Earlier quoted context omitted.

If you're waiting for this to be the straw to move away from Wells Fargo after the last set of scandals, you may have some issues...

Yep, the California AG basically said "Wells Fargo is a criminal enterprise masquerading as a corporation" and I'm inclined to believe him.

I can't find a quote like this -- is there any chance you have the source?

Re: Tell HN: Wells Fargo completely offline

#186

Post by shagie »

pushtheenvelope wrote: post #159
surge wrote: post #115

Earlier quoted context omitted.

They had redundancy, the redundancy failed. I mean truthfully when do you get to test your redundancy against a true disaster. It was a mess. WF is 20 companies rolled into one so the fact the disparate systems from 10 different banks works at all is kind of a miracle.

Facebook regularly takes down multiple data-centers at a time to stress test this. Its users rarely notice (which I think is the point).

There is a different level of criticality for "my post didn't go through, hit refresh" and "my transaction didn't go through - the restaurant said my card isn't working."

Would you honestly want to go to a bank and say "if we unplug this, we can find out what fails."

Re: Tell HN: Wells Fargo completely offline

#187

Post by shagie »

bitxbitxbitcoin wrote: post #17
jwcooper wrote: post #8

Earlier quoted context omitted.

Local fire Department commented on their facebook page [1] that dust from construction triggered the fire suppression system and not a fire. [1] https://www.facebook.com/LakeJoFD

The next worst thing for electronics after fire - water! Or are there non liquid fire suppression systems such as pulling all the oxygen out of a room (if there aren't any people in it)?

Ahh... The VAX war. http://bogpeople.com/funny/content/vaxwarstory.html

> VAXen, my children, just don't belong some places.

Suffice to say... yes. There are fire suppression systems that will pull oxygen out of the room. People are advised to leave the room before the fire suppression system takes effect.

Re: Tell HN: Wells Fargo completely offline

#188

Post by icedchai »

CydeWeys wrote: post #55
WrtCdEvrydy wrote: post #13

Earlier quoted context omitted.

If you're waiting for this to be the straw to move away from Wells Fargo after the last set of scandals, you may have some issues...

Unfortunately they're the holders of my mortgage (it was sold twice and they ended up with it). So I don't even have a choice of being their customer for the next ~25 years.

If you care that much, you can refinance.

Re: Tell HN: Wells Fargo completely offline

#189

Post by closeparen »

surge wrote: post #115
mabbo wrote: post #12

Earlier quoted context omitted.

Incredible. They had all their mission-critical infrastructure in a single data center. How many billions of dollars do they make per year? And they can't afford even a tiny bit of redundancy? If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.

They had redundancy, the redundancy failed. I mean truthfully when do you get to test your redundancy against a true disaster. It was a mess. WF is 20 companies rolled into one so the fact the disparate systems from 10 different banks works at all is kind of a miracle.

We run that drill every Wednesday morning.

Re: Tell HN: Wells Fargo completely offline

#190

Post by computerex »

hinkley wrote: post #162
surge wrote: post #115

Earlier quoted context omitted.

They had redundancy, the redundancy failed. I mean truthfully when do you get to test your redundancy against a true disaster. It was a mess. WF is 20 companies rolled into one so the fact the disparate systems from 10 different banks works at all is kind of a miracle.

I can't recall who it was, but there was a big outage due to a data center running a monthly generator test and having a major customer go dark. The data center had 7 generators for 4 server rooms: 1 per room, a backup for each pair of rooms, and a backup for the backups. The primary and then both backups failed, so out went the lights. You're pretty much damned if you do and damned if you don't. If you touch things…

Look at Netflix with chaos monkey and simian army. Netflix routinely does catastrophic destructive testing on their production systems, sometimes evacuating an AWS region entirely. To them, a server going down is a non-event, because they designed their systems with the premise that servers go down.
Post reply on HN