381–390 of 535 posts
For the many of us who have built businesses dependent on S3, is anyone else surprised at a few assumptions embedded here? * "authorized S3 team member" -- how did this team member acquire these elevated privs? * Running playbooks is done by one member without a second set of eyes or approval? * "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years" The…
Earlier quoted context omitted.
I agree testing and automation are good. I think they need to go beyond this to formal verification, for something on this scale and reliability. NASA doesn't make these sorts of mistakes. By the way - this is not just Amazon's problem now. We know the internet has a single point of failure. So does a lot of IoT. When will we experience the first Suicide DevOps?
NASA doesn't make these sorts of mistakes https://www.wired.com/2010/11/1110mars-climate-observer-repo...
Earlier quoted context omitted.
Which is fine until those regions go down. A status page, in my mind, should have a fallback on a completely different service provider.
Exactly. What they should do is have the status system be independent of AWS so it can report issues regardless of AWS service status.
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".
Earlier quoted context omitted.
Note: they say "S3 is DESIGNED for 11 9s of durability". It's PR-speak to say that they don't give you any guarantee, but in theory the system is designed in a magnificent way.
11 9s of durability is about the likelyhood of AWS loosing your data. It doesn't cover the likelyhood of you being able to access your data that's called availability. For example on GCS (Google's S3)...A storage class specifies how many locations the data is made available. All storage classes share the same durability (chance of google loosing your data) of 99.999999999%, but have different availability (chance of…
Earlier quoted context omitted.
Note: they say "S3 is DESIGNED for 11 9s of durability". It's PR-speak to say that they don't give you any guarantee, but in theory the system is designed in a magnificent way.
11 9s of durability is about the likelyhood of AWS loosing your data. It doesn't cover the likelyhood of you being able to access your data that's called availability. For example on GCS (Google's S3)...A storage class specifies how many locations the data is made available. All storage classes share the same durability (chance of google loosing your data) of 99.999999999%, but have different availability (chance of…
Earlier quoted context omitted.
Note: they say "S3 is DESIGNED for 11 9s of durability". It's PR-speak to say that they don't give you any guarantee, but in theory the system is designed in a magnificent way.
11 9s of durability is about the likelyhood of AWS loosing your data. It doesn't cover the likelyhood of you being able to access your data that's called availability. For example on GCS (Google's S3)...A storage class specifies how many locations the data is made available. All storage classes share the same durability (chance of google loosing your data) of 99.999999999%, but have different availability (chance of…
git commit -m 'typo'
All those tweets saying "turn it off and back on again"?
"We accidentally turned it off, but it hasn't been turned it off for so long it took us hours to figure out how to turn it back on."
Poorly-presented jokes aside, this is rather concerning. The indexer and placement systems are SPOFs!! I mean, I'd presume these subsystems had ultra-low-latency hot failover, but this says they never restarted, and I wonder if AWS didn't simply invest a ton of magic pixie dust in making Absolutely Totally Sure™ the subsystems physically, literally never crashed in years. Impressive engineering but also very scary.
At least they've restarted it now.
And I'm guessing the current hires now know a lot about the indexer and placer, which won't do any harm to the sharding effort (I presume this'll be being sharded quicksmart).
I wonder if all the approval guys just photocopied their signatures onto a run of blank forms, heheh.