Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Wells Fargo completely offline

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 256 posts

Re: Tell HN: Wells Fargo completely offline

#161

Post by gdubs »

40acres wrote: post #153
habosa wrote: post #141

For those who have worked in bank IT, I have some questions. So these days there's very little physical money. Most of my "wealth" is just entries in various digital ledgers. My bank says I have $XXX and my brokerage says I have $YYY and my retirement account says I have $ZZZ. Let's go with the bank account case. Is it possible that a catastrophic accident or attack could wipe my balance down to $0 with no way to rec…

Generally I think here is where the FDIC and SPIC would come in.

I’d love someone who knows the details to explain how that claim process would work, because while the FDIC would insure in the event a bank goes under, in that scenario there’s usually records.

What would happen if there were no records? Surely there’d be lots of people making significant deposits in between snapshots. If you’re “paperless” I’m not even sure how you’d reconstruct your balance.

Would certainly be time consuming, and in this case time would certainly be money.

Re: Tell HN: Wells Fargo completely offline

#162

Post by hinkley »

surge wrote: post #115
mabbo wrote: post #12

Earlier quoted context omitted.

Incredible. They had all their mission-critical infrastructure in a single data center. How many billions of dollars do they make per year? And they can't afford even a tiny bit of redundancy? If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.

They had redundancy, the redundancy failed. I mean truthfully when do you get to test your redundancy against a true disaster. It was a mess. WF is 20 companies rolled into one so the fact the disparate systems from 10 different banks works at all is kind of a miracle.

I can't recall who it was, but there was a big outage due to a data center running a monthly generator test and having a major customer go dark. The data center had 7 generators for 4 server rooms: 1 per room, a backup for each pair of rooms, and a backup for the backups. The primary and then both backups failed, so out went the lights.

You're pretty much damned if you do and damned if you don't. If you touch things that are working you could break them. If you don't touch things you never know what'll happen and you get fewer opportunities to learn. Move your servers around geographically and you might improve the odds that anything is working by reducing the odds that everything is working.

I don't think we're quite to a place yet where having servers down can be characterized as a non-event. Even if the customer can't see a behavioral difference, business units still tend to get quite anxious, and sometimes their theatrics put the whole process in jeopardy (not unlike trying to rescue a drowning man). It just hasn't been normalized yet.

Re: Tell HN: Wells Fargo completely offline

#163

Post by jen729w »

pushtheenvelope wrote: post #159
surge wrote: post #115

Earlier quoted context omitted.

They had redundancy, the redundancy failed. I mean truthfully when do you get to test your redundancy against a true disaster. It was a mess. WF is 20 companies rolled into one so the fact the disparate systems from 10 different banks works at all is kind of a miracle.

Facebook regularly takes down multiple data-centers at a time to stress test this. Its users rarely notice (which I think is the point).

They're I've worked for banks here in Australia. Everything is 30+ years old. It's a shambles.

Re: Tell HN: Wells Fargo completely offline

#164

Post by hinkley »

rolph wrote: post #149

OK, Hanlon's razor applies here: https://en.wikipedia.org/wiki/Hanlon%27s_razor recall Stuxnet, its possible this was an attack, somesort of malicious mod to firmware, but_ Hanlon's razor. from the reddit: "throwawayfordays75 1399 points 8 hours ago*2 Throwaway since I have first hand knowledge. Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the net…

That user posted an update:

"everything minus core network gear was manually being unplugged from any PDUs to help the control the initial power-on."

Can't we... Why don't we have rack hardware that can handle this situation? I thought some HDD RAID solutions had circuitry to keep them from browning themselves out while spinning up the disks. I guess I'm surprised this isn't a solved problem at the rack level now.

Or have we been so focused on never cold booting a rack of servers that we haven't spent any effort on foolproofing of cold booting a rack of servers?

[Edit: answering my own question] apparently these exist and are called Managed PDUs. Can we deduce WF doesn't have them?

Re: Tell HN: Wells Fargo completely offline

#165

Post by Rapzid »

nostrademons wrote: post #93
mywittyname wrote: post #68

Earlier quoted context omitted.

Lots of old companies are like this. They don't invest in critical infrastructure because it would bring them below their 20% margins.

Wells is weird though. When I started banking with them in 2009, they were ahead of the curve in online infrastructure - it was much easier to access them online than any of the banks I was used to back in Massachusetts. They also had a reputation for being both honest and conservative - they were the only AAA-rated bank in 2007, they were #1 in green rankings, and Warren Buffett had invested in them because of their…

Wells was one of the earlier companies to have the CTO report directly to the CEO.

All the crazy sales numbers and bogus account shenanigans were going on back in 2003/2004 when I worked there. I ratted out more than one professional banker to branch managers and up over that crap. A fun one was the home equity lines people would open without customer knowledge and link up to overdraft protection. The customer would never owe, nor know, anything until one day an overdraft hit their equity line, and then they got notified of late payments.. I don't miss working for a Bank.

Re: Tell HN: Wells Fargo completely offline

#166

Post by Klathmon »

40acres wrote: post #153
habosa wrote: post #141

For those who have worked in bank IT, I have some questions. So these days there's very little physical money. Most of my "wealth" is just entries in various digital ledgers. My bank says I have $XXX and my brokerage says I have $YYY and my retirement account says I have $ZZZ. Let's go with the bank account case. Is it possible that a catastrophic accident or attack could wipe my balance down to $0 with no way to rec…

Generally I think here is where the FDIC and SPIC would come in.

I can't even imagine the hellish process that you would have to go through to prove that you actually owned the money you did to receive the FDIC or SPIC pay.

I'm genuinely curious what the process would look like, if it even applies. Where would you even start if the bank just suddenly says "you have $0"?

Re: Tell HN: Wells Fargo completely offline

#168

Post by rolph »

hinkley wrote: post #164
rolph wrote: post #149

OK, Hanlon's razor applies here: https://en.wikipedia.org/wiki/Hanlon%27s_razor recall Stuxnet, its possible this was an attack, somesort of malicious mod to firmware, but_ Hanlon's razor. from the reddit: "throwawayfordays75 1399 points 8 hours ago*2 Throwaway since I have first hand knowledge. Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the net…

That user posted an update: "everything minus core network gear was manually being unplugged from any PDUs to help the control the initial power-on." Can't we... Why don't we have rack hardware that can handle this situation? I thought some HDD RAID solutions had circuitry to keep them from browning themselves out while spinning up the disks. I guess I'm surprised this isn't a solved problem at the rack level now. Or…

it would seem to be a possibility. im also thinking 10 finger power up sequence, as in: insert power cord 1 into powersocket 1, wait till audible beep, insert power cord 2 into power socket 2, wait till audible beep etc. :-D

Re: Tell HN: Wells Fargo completely offline

#169

Post by peter_d_sherman »

patio11 wrote: post #120
driverdan wrote: post #97

Earlier quoted context omitted.

I signed up for Stripe in 2011 or 2012. About six months after I signed up an account rep from Wells Fargo called me asking about my merchant account. Stripe never made it quite clear what was going on but apparently in the early days they opened accounts at Wells Fargo on your behalf without telling you.

I also signed up for Stripe in 2011. The Terms of Service mentioned Wells, beginning in the first paragraph. You might want to look in your files and/or the Wayback Machine. https://web.archive.org/web/20111017081436/https://stripe.co...

Speaking as a Law afficionado, that is one great Terms of Service document! Kudos to whichever legal team wrote it! I'm not saying that it completely anticipates all potential customer-company interactions that could occur (no legal document could), but it seems to go very far, and seems to show a good understanding of the many potential issues that can arise. My future company might use this document (and others) for inspiration on how to craft its own Terms of Service documents. Thanks for the link!

Re: Tell HN: Wells Fargo completely offline

#170

Post by reilly3000 »

clairity wrote: post #75

for those who haven't taken the plunge, now is a good time to move to a credit union, regional bank, or online bank: - for instance, in cali: https://www.golden1.com/ - high interest accounts: https://www.bankrate.com/banking/checking/best-checking-acco... - good rate online bank: https://empower.me/ get better service, lower/no fees, and good convenience without all the economy-damaging selfishness and hubris.

Is there any decent bank/CU with a real reporting API?
Post reply on HN