Bloomberg does it once a year. That said, I have yet to encounter a company more obsessed with business continuity. I don't doubt that their failover systems and testing of them are well beyond the typical.
I think Bloomberg can stand to be down for 8 hours to simulate a disaster. Banks with legacy systems and people constantly dependent on them to conduct business can't risk an actual incident happening because they were testing what would happen if an incident happened. Netflix designed their stuff from the ground up to fail over. Large monolith corporations who've inherited systems from other companies they've bought…
Business critical systems can’t afford not to test failover.
You can bail out of a test at the first sign of trouble. When a real outage hits, there’s no telling how long it will take to recover.
Facebook regularly takes down multiple data-centers at a time to stress test this. Its users rarely notice (which I think is the point).
There is a different level of criticality for "my post didn't go through, hit refresh" and "my transaction didn't go through - the restaurant said my card isn't working." Would you honestly want to go to a bank and say "if we unplug this, we can find out what fails."
Facebook handles money too, though. Also I think the parent was making the point that Facebook builds software with resiliency in mind so when a failure does happen, the software deals with it gracefully.
OK, Hanlon's razor applies here: https://en.wikipedia.org/wiki/Hanlon%27s_razor recall Stuxnet, its possible this was an attack, somesort of malicious mod to firmware, but_ Hanlon's razor. from the reddit: "throwawayfordays75 1399 points 8 hours ago*2 Throwaway since I have first hand knowledge. Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the net…
That user posted an update: "everything minus core network gear was manually being unplugged from any PDUs to help the control the initial power-on." Can't we... Why don't we have rack hardware that can handle this situation? I thought some HDD RAID solutions had circuitry to keep them from browning themselves out while spinning up the disks. I guess I'm surprised this isn't a solved problem at the rack level now. Or…
Some servers "stage" power on to disks. as spinning up disks is often the largest source of inrush current for e.g. a 10 disk server might only apply power to bays 2 at a time with a delay between each, this doesn't solve the inrush current problem of switching on a rack full or servers together (where managed PDUs come in)
Unfortunately they're the holders of my mortgage (it was sold twice and they ended up with it). So I don't even have a choice of being their customer for the next ~25 years.
Refi that bad-boy to your local credit union
Can't. It's a fixed 30 year mortgage at a 3.375% rate. Refinancing it will increase that by at least a point. Plus, I originally did get the loan through a local credit union, who then sold it to a consolidator who sold it to a big national bank (which happened to be Wells Fargo). Local credit unions generally don't hold onto mortgages. They don't have the economies of scale to service them efficiently.
Unfortunately they're the holders of my mortgage (it was sold twice and they ended up with it). So I don't even have a choice of being their customer for the next ~25 years.
As edoceo says below, I recommend refinancing with a credit union of your choice, preferably one that will pledge in writing to service the loan for its life.
Rates have gone up since, unfortunately. I'd be paying thousands in refi fees for the privilege of paying more each month in interest. No thanks.
For everyone complaining about "how could they not have failover" let me ask you this: Would you take a job with Wells Fargo to fix their infrastructure? And if you did take such a job, how long do you think it would take you to get the budget and approvals from all the auditors necessary to fix everything? How much would they have to pay you to take that job, knowing how frustrating it would be to get anything done?…
I can't recall who it was, but there was a big outage due to a data center running a monthly generator test and having a major customer go dark. The data center had 7 generators for 4 server rooms: 1 per room, a backup for each pair of rooms, and a backup for the backups. The primary and then both backups failed, so out went the lights. You're pretty much damned if you do and damned if you don't. If you touch things…
Look at Netflix with chaos monkey and simian army. Netflix routinely does catastrophic destructive testing on their production systems, sometimes evacuating an AWS region entirely. To them, a server going down is a non-event, because they designed their systems with the premise that servers go down.
Netflix is a great example of how to create robust systems. Keep in mind though, they have a very different risk profile than a large bank. no one is going to lose their life’s savings if netflix entire infrastructure crumbles. This might happen in a bank with a single server outtage. Don’t get me wrong, if I start a bank today, I use chaos monkey stategy. But if I take over a bank infrastructure, with cobol still representing a huge percentage of mission critical code that everyone is scared to touch... no chaos monkey. I might deliberately turn off a server for an hour to see what chaos ensues, but it’ll be after 3 months of analysis, longer if I begin to suspect the system does more than anyone can remember.
For everyone complaining about "how could they not have failover" let me ask you this: Would you take a job with Wells Fargo to fix their infrastructure? And if you did take such a job, how long do you think it would take you to get the budget and approvals from all the auditors necessary to fix everything? How much would they have to pay you to take that job, knowing how frustrating it would be to get anything done?…
Unfortunately they're the holders of my mortgage (it was sold twice and they ended up with it). So I don't even have a choice of being their customer for the next ~25 years.
This is why I like the Canadian system of mortgages. I have a 30 year mortgage, but every 5 years I have to "renew" it. At that time, I have to renegotiate the rate for the next 5 years. As part of this negotiation, I can just switch banks if I want. Or to a private lender. Or to anyone really. It seems weird to me that you are beholden to an entity you've never signed any contract with.
Is there something to prevent the rate from skyrocketing when you renew the mortgage? Doesn't this have all the issues people occasionally have with ARMs?