Earlier quoted context omitted.
They should have asked AI to implement the redundancy.
How would have that changed anything?
First of all they could write an amazing slop blog PR post about it afterwards
AWS says it can't restore some data from mideast facilities struck by Iran
381–390 of 468 posts
Re: AWS says it can't restore some data from mideast facilities struck by Iran
#382Earlier quoted context omitted.
Especially with Amazon, who are well known for squeezing every last bit of profit from their employees, contractors etc., it doesn't really sound surprising. "Offsite backups?! Sure, you could have had that if you had found the right page in the AWS console and if you would have paid 50% extra!"
Oh sorry, you picked "AWS Backup" but you actually needed to use "Backup AWS" to solve that problem. Perhaps hop on a call with our sales engineering and cost magnification teams to guide you to a better, more solutioned, tomorrow?
I sounded all legit until the comma after "solutioned". I suspect that might be a fake.
Re: AWS says it can't restore some data from mideast facilities struck by Iran
#383My personal bet is there's an 80% chance this is caused by some internal bootstrapping problem that they've messed up. AIUI all the main cloud vendors are in trouble here. The automation project I work on is expressly designed to help folks solve this DR/bootstrapping problem. Soo many people get this wrong. Of course missiles don't help things, but I'd bet AWS is primarily to blame here. I'd love an actual technical…
I work on the same kind of thing, and while we think hard about bootstrap problems, we always find new surprising ones. The problem is you never know until you do it, and creating a faithful test of restarting giant systems is economically impossible. Because if you say to the boss, "look, I need 1 million now to test against a maybe 100 million loss, maybe in 10 years" they don't give you the money (and rightly so).
Re: AWS says it can't restore some data from mideast facilities struck by Iran
#384Re: AWS says it can't restore some data from mideast facilities struck by Iran
#385Re: AWS says it can't restore some data from mideast facilities struck by Iran
#386Earlier quoted context omitted.
The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.
its also the fact that a lot of fundamental systems work in trade offs. Do you want performance, or correctness. Well, if you want performance you use write through caching and in the case of distributed storage: more nodes confirming the block before returning. Huge performance cost. Outsourcing this just means someone else makes these tradeoffs, they will prioritise the general case- and they’re even more incentivi…
if you want correctness* you use write through caching.
My mistake, if you want performance you choose write-back caching, and fewer nodes need to acknowledge the write. Sorry for clumsily typing the inverse when I was in a morning haze waking up :(
Re: AWS says it can't restore some data from mideast facilities struck by Iran
#387This interview with an AWS leader isn’t aging well, from CBS Sunday morning: Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!…
There's something reassuring in this for me. There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle…
AWS has a history of making hand wavy explanations on the robustness of its infrastructure that were misleading at best. See comments about how regions are truly independent only for everyone to find out if us-east-1 goes down you could still be down even if you built in other regions. Those dependencies were not well documented and previously hand-waved away by AWS when it boasted about how its regions were truly independent.
Similar here, there’s a lot of detail that got hand-waved away by a sloppy “yeah we good” puff PR answer.
Re: AWS says it can't restore some data from mideast facilities struck by Iran
#388Earlier quoted context omitted.
I think it applies to all humans of all ages in one way or another, these varying imperfections is what makes us what we are
its not true. If everyone actually did the bare minimum, everything would screech to a halt immediately. In fact people doing that imo is part of what causes the decline of empires. You can, in fact, trust people further than you can throw them. Its a cousin of the mindset that the reason people don't steal is because they think they will be caught and rationally weigh up based on the value they gain and the chance o…
One needs the state, the other needs intrinsic morals. One is a tax on all actions, the other forms working societies.
Re: AWS says it can't restore some data from mideast facilities struck by Iran
#389Earlier quoted context omitted.
I'm saying that if you have eliminated more mundane failures like dying harddrives, cosmic rays and so on from your systems and your calculation ends up with 11 nines then actually those "force majeure" events are probable enough that they dominate whatever other residuals are supposedly hiding in those last 0.0000000001%. The region has seen a bunch of wars in the last 100 years, so the annual war-rate is > 1%. Even…
You don’t want to blend probabilities like this, because the tactics you use as a consumer vary between the two. If you consider 11 9s like “object AFR”, you might build systems that are resilient to very occasional single object loss. And it’s useful to know at what rate that might occur. Whereas with these force majeure events you’d want a complete DR setup, and it’s typically an async recovery. Here it is useful t…
Seems like begging the question to me. You can't blend the numbers because amazon didn't blend the numbers.
If they did and miraculously still arrived at 11 9s then that would also cover things such as wars and natural catastrophes, e.g. because they do offsite backups internally.
Re: AWS says it can't restore some data from mideast facilities struck by Iran
#390Ahhh the downside of “data sovereignty” rules and the de-globalization meme strikes again. I’d bet a bajillion dollars the reason this happened is due to government thinking it’s a good idea to make it illegal to store data outside their country. Hey EU, take note of this next time you create silly data residency requirements that don’t allow data to travel outside your region. Encryption is an easy solution to multi…
The EU is quite big; there are seven AWS regions there, each with three AZs.