"It is cheaper to design a system that must be up which accounts for a data center being totally down and a portion of the system being totally unavailable than to add more datacenter mitigations." Citation needed - the same issue with testing, data races and expensive bandwidth come up.
At high energy the lead time for the components is measured not in days but in years.
And so is development time of any distributed software system, and training time required to operate it correctly
Me: Hesitation at last job moving absolutely everything (including backups) to AWS because if it goes down it's a problem I'm a firm believer in some kind of physical/easily accessible backup. Coworkers: "You're an f'n idiot. Amazon and Facebook don't go down, you're holding us back!" Me: leaves cause that treatment was the final straw Amazon and Facebook both go down within a month of each other, and supposedly they…
AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs. Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZ…
> No point in investing in S3 and then doing it again yourself. I mean that's just obviously wrong, though. There is a point. > Either you don't trust the cloud and you can run NAS or equivalent (with s3 APIs easily today) much cheaper or trust them to keep your data safe and available. What if you trust the cloud 90%, and you trust yourself 90%, and you think it's likely that the failure cases between the two are li…
My point is if your redundancy is better than AWS then why pay for them ? If it not they why invest in your own?. You can argue that you protect against different threats than AWS does . So far I have not seen a meaningful argument of threats a on Prem protects differently than the cloud that you need both . Say for example your solution is to put all your data backups on the moon then it makes sense to do both, AWS…
If you trust your airbag, why bother with the sealtbelt?
True with two independent servers at 90% each, that’s 0.1^2 = 1% chance both fail— so redundancy can add a lot of reliability.
You assume failures are uncorrelated. Which, depending on what you think you are protecting yourself from, might or might not be true. (Consider a buggy software release which incorrectly deletes a backup. Depending on the bug it’s very possible it will delete in both places.)
If one buggy software release can delete both copies, then you don't have actual redundancy from the point of view of that issue.
Sometimes, you have a component which fails in such a way that your redundancies can't really help. I once had to prepare for a total blackout scenario in a datacenter because there was a fault in the power supply system that required bypassing major systems to fix. Had some mistake or fault happened during those critical moments, all power would've been lost. Well-designed redundancy makes high-impact incidents less…
To my mind, among the more frustrating aspects to implementing protection against failure is that the mechanisms to be added can themselves cause failure. It's turtles all the way down.
You need to pick your battles and choose what you want to protect against to mitigate risk and enable day-to-day operations.
For example, too often people will set up clustered databases and whatnot because "they need HA" without much thought about all the other potential effects of using a cluster, such as much more complicated recovery scenarios.
In the vast majority of cases, an active-passive replicated database with manual failover is likely to have fewer pitfalls and gives you the same operational HA a clustered database would, even though in the case of a (rare) real failure it would not automatically recover like a cluster might.
I'd be surprised if they needed backups for a few hours of downtime with (reportedly) complete recovery where no data was corrupted. There are industries where this would be required, and it's possible I guess, but neither of these downtime events were "data loss" events, just availability events for short-ish periods of time that wouldn't - for me - result in activating our DR plans. I must admit that I do always tr…
AWS has had at least one documented incident where a region had an S3 failure that was not recoverable. They lost about 2% of all data. That might not sound like much but if you have a lot of data, partial restoration of that data doesn't necessarily leave your system in a functional state. If it loses my compiled CSS files I might be able to redeploy my app to fix it. Then again if I'm a SaaS company and that file w…
Which incident is this? I can’t find it online. The closest I can recall is when they lost some number of EBS volumes. We were affected by that, but ran snapshots (to s3) to recover the affected servers.
I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you…
> You need a back up generator and to be a short distance away from a petrol station My building has a natural gas backup generator.
Does it have its own gas well? My sister has a home backup generator, they lost power during some cold snap and some pumping component failed and her neighborhood lost gas too. The only house in the neighborhood that had heat/power had a big propane tank because it was built before the neighborhood got gas.
I’ve never seen a data center with natural gas backup power. But I don't know if that's because of reliability or if it's too expensive for a big natural gas hookup that's used rarely. Though I have heard of the opposite -- using natural gas turbines as primary power and utility power as backup.
Now that everyone and their dog is on AWS, it is not just 'a website stops working', half the world, from telephones to security doors and Iot equipment, stops working? I am not sure if the movement the cloud has reduced amount of failures, but it definitely has made these failures more catastrophic. Our profession is busy makin the world less reliable and more fragile, we will have our reconning just like the shippi…
all I've noticed is slack was a bit unreliable for a little bit, but i just carried on and otherwise ignored it. my world did not stop working.
My apartment block has a dialing system, that, instead if using a cale that goes to your apartment, relies on IP telephony and calls your mobile phone. It stos working if there is no internet, or your phone is out of battery, or you are not home but your wife is.
I've had way more networking and availability failures from Hetzner this year alone then I've ever seen from AWS. They regularly replace their networking switches without any redundancy, leaving entire DCs offline for hours. They're okay for hobby projects, but I would never host a business-critical site with them
Cannot confirm, do you have details? Yes, Hetzner upgrades DCs (datacenter buildings), but they are the equivalent to AWS AZs (Availability Zones). When they upgrade a DC, they notify way in advance, and if you set up your services to span multiple DCs as is recommended, it does not affect you. We run high-availability Ceph, Postgres, and Consul, across 3 Hetzner DCs, and have not had a Hetzner-induced service downti…
That's fair enough, I was comparing single-AZ AWS outages to single-DC Hetzner outages, since that seems to be what people are focusing on. For multi-DC deployments, I think laumars' sibling response to mine makes a much better argument—ultimately, you're still choosing who to pay and who to trust, and if things go down, there's nothing you can do to fix it. "Low-tech" cloud providers like Hetzner, Colo providers, amazon, PaaS—in a physical downtime event like this one, they're all the same.