Human is complicated. Let them relax and let the androids do the boring jobs :D
Summary of the Amazon S3 Service Disruption
471–480 of 535 posts
Re: Summary of the Amazon S3 Service Disruption
#472Earlier quoted context omitted.
A complete restart of the index subsystem would require downtime. Note: they are not saying those servers have never been restarted - it's highly likely they get restarted regularly. But, a complete restart of the index subsystem implies that you shut everything down first and restart it all at once, which is what was forced to happen two days ago.
Why can't the index subsystem itself have a backup then? When the primary subsystem is being restarted/rebuilt, the secondary takes over.
thats kind of like asking why git doesn't have a single backup... it's a distributed system, there's not just one backup, there are lots of little partial backups.
Re: Summary of the Amazon S3 Service Disruption
#473Earlier quoted context omitted.
"While removal of capacity is a key operational practice, in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level. This will prevent an incorrect input from triggering a similar event in the future."
I wonder as to why this wasn't thought of while creating the system? Of course, I don't have experience at that scale, so just wondering.
It seems like sometimes this is just how iteratively automating things works, especially on an internal-facing tool.
You have some process that starts out being "deploy this app with this java code". You deploy once and while, so it's not a big deal. But then those changes get a bit more frequent and so you pull out the common bits and the process becomes "make this YAML change in git and redeploy the app".
That works until you find yourself deploying 5 times a day, so you turn it into a MySQL table, and the process becomes "write a ROLL plan that executes this UPDATE x=y WHERE t=u; command"
After a while you get super annoyed at some quirk of the commands and figure, "Ok, fine, I'll just add an endpoint and some logic that just does this for the command case."
Then you wanna go on vacation and the new guy messed up the API request last week, so you figure, "I'll just add a little JS interface with a little red warning if the request is messed up in this way or that before I go".
You get back from vacation and some original interested party (whoever has wanted all these changes deployed) watched the intern make the change and thinks they could just do it themselves if they had access to the interface. You're wary, but you make the changes together a few times and maybe even add a little "wait-for-approval" node in the state machine.
Life is good. You've basically de-looped yourself, aside from a quick sanity check and button press, instead of what was a ~2 hour code + build + PR + PR approved + deploy process.
Then that interested party goes to work for Uber and the rest of your team adds a few functionalities on top of the interface you built and it all goes pretty well, until you realize that now that this thing that used to be 20 YAML objects is now 50k database records, and a bunch of them don't even apply anymore. So you build a button to disable some group of them, but after getting it deployed you realize it's actually possible to issue a "disable all" request accidentally if you click a button in your janky JS front-end before the 50k records download and get parsed and displayed. Oops! This mistake that you and the original interested party would have never made (because you spent the last 2 years thinking about all this crap) is probably a single impatient anxious mouse-click away from happening. So you make a patch and deploy that.
Congrats! You found that particular failure mode and added some protections for it, and maybe added some other protections like rate-limiting the deletions or updates or whatever. That's cool, but is that every failure mode? I bet it isn't. What happens when someone else thinks you have too many endpoints and just drops to SQL for the update?
Basically, yeah, of course you think of this stuff while iterating on it. But you figure "only power users are on the ACL" or "my teammates will understand the data model before making changes, or ask me first" or "that's what ROLL plans are for" or "I'll show a warning in the UI" or whatever. Fundamentally, you're thinking about a way to do a thing, if you're even thinking about it at all.
So yeah, that's what I've spent the last year or two doing. :-)
Re: Summary of the Amazon S3 Service Disruption
#474> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
It is always going to be like that. If you write software that has a rule "do not remove more than 5% of capacity at once" it will always work, yet if you tell a systems engineer please do not remove 5% of the capacity at once it fails with 0.0x% chance. The solution is that you move the execution of change into a system that spits out steps that is automatically executed by the system itself, entirely removing the human factor.
Re: Summary of the Amazon S3 Service Disruption
#475Earlier quoted context omitted.
It is unreasonable for me to think that company owners should have the spine to say, "We take the decision to fire someone very seriously. We'll take your comments under consideration, but we retain sole discretion over such decisions." It irks me that businesses fire people because of pressure from clients or social media. But having never been the boss, I may be missing something.
Historically some cultures practiced mock firing as a way to appease an angry customer. This was back in the day when most business transactions occured face to face so the owner should demand the employee to pack their belonging and leave the premises in full view of the customer. Of course this is all for show but this kind of public humiliation seems to satisfy even the most difficult customers.
Even when the customer knew it was show, I see it as a way of saying, "yes, we acknowledge that we screwed up and make a public, highly visible note of it that will be recorded in the annals of peoples' gossip in this area".
Re: Summary of the Amazon S3 Service Disruption
#476Earlier quoted context omitted.
I wonder as to why this wasn't thought of while creating the system? Of course, I don't have experience at that scale, so just wondering.
It seems like sometimes this is just how iteratively automating things works, especially on an internal-facing tool. You have some process that starts out being "deploy this app with this java code". You deploy once and while, so it's not a big deal. But then those changes get a bit more frequent and so you pull out the common bits and the process becomes "make this YAML change in git and redeploy the app". That work…
Even I have been doing this, but not quite at this scale, it is mostly python scripts to automate something, but because of the low scale and that I am the sole owner + user, I am good to go :-D
Re: Summary of the Amazon S3 Service Disruption
#477Earlier quoted context omitted.
You don't have to know what would be a mistake. E.g. if the tool is used most of the time to operate on a small set of servers, you have some extra confirmation or command-line option for removing a large set. That's good UI design in tools with powerful destructive capabilities. You make the UI to do lots of things v.s. the few things you do routinely different enough that there's no mistaking them.
You can also have the program tell the user what's going to happen (if it can be computed beforehand), e.g. "This will affect 138 server(s)."
This doesn't work. Users learn to ignore the message.
Re: Summary of the Amazon S3 Service Disruption
#478Earlier quoted context omitted.
But that still allows you to make a mistake - by pressing y when that's the wrong thing to do.
There's a balance to be struck. I'd say number of hoops you have to jump through to do something should scale with the potential impact of an operation. That said, the only way to completely prevent mistakes is to make the tool unable to do anything at all. (Or to encode every possible meaning of the word "mistake" in your software. If you could do that, you would probably get a Nobel prize for it.)
We're assuming that the software in question is even aware of the potential impact. It might not have that information.
Re: Summary of the Amazon S3 Service Disruption
#479Earlier quoted context omitted.
One reason to like a facet of Japanese management culture: if a customer wants someone to rake over the coals you offer management, not employees. Internal repercussions notwithstanding, externally the company is a united front. It cannot cause mistakes by luck, accident, or happenstance, because the world includes luck, accidents, and happenstance, so any user-visible error is ipso facto a failure of management.
apparently this is (or was) a job in japan. companies would hire what amounts to an actor to get screamed at by the angry customer, and pretend to get fired on the spot. rinse, repeat whenever such appeasement is required.
Well, well, well...scribbles note into Trello: "Today I got fired, it was all my fault, our poor customers suffered" Blog posts as a service.
Re: Summary of the Amazon S3 Service Disruption
#480Earlier quoted context omitted.
> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well. Exactly. It seems likely that Amazon tests the restart operation, but it would be hard to test it at full us-east-1 scale. Running a full S3 test cluster at that scale would likely be a prohibitive expense. Perhaps the "index subsystem" and "placement subsystem" are small enoug…
> Failure at every level has to be simulated pretty often to understand how to handle it Keep in mind, S3 "fails" all the time. We regularly make millions of S3 requests at my work. Usually we get 1:240K failure rate (mostly GETs), returning 500 errors. However, if you're really hammering an S3 node in the hash ring (e.g. Spark job), we see failures in the 1/10K range, including SocketExceptions, where the routed IP…
If you retry the get does it succeed usually?