51–60 of 535 posts
For the many of us who have built businesses dependent on S3, is anyone else surprised at a few assumptions embedded here? * "authorized S3 team member" -- how did this team member acquire these elevated privs? * Running playbooks is done by one member without a second set of eyes or approval? * "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years" The…
It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level."
At the organizational level, I guess it wasn't rated as all that likely that someone would try to remove capacity that would take a subsystem below its minimum. Building in a safeguard now makes sense as this new data point probably indicates that the likelihood of accidental deletion is higher than they had estimated.
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Sounds like an opportunity for machine learning. Anyone want to write an AI BOFH?
This is the bit that'd worry me most; you'd think they'd be testing this.
Earlier quoted context omitted.
That person is the least likely to make that mistake again!
No! Off with this head!!! On a more serious note, if you've never done something like this, you haven't had enough interesting projects. I've had a decent career and I still managed to: * re-deploy the current application version in all our data centers, instead of the new version, in a period when our deployment wasn't a 0-downtime one * rename all the Jenkins jobs on the server to the same name, thus deleting hundr…
1. No organization anywhere is a paragon of excellence, and everyone can benefit from improvement. 2. Every organization is made up of humans just like you. With all that entails.
Some things which seem blatantly obvious after the fact are easily overlooked when the pressure to deliver is high and other issues are taking precedence.
I completely agree with your statement. In fact, when I do interviews, one of my favorite and most insightful questions to ask is, basically, "tell me about a time you screwed the pooch." If they don't have a story and they worked in ops, then it can suggest they didn't really do much. The really sharp ones I've interviewed have a good story or two (and can tell it in excruciating detail. =)
* At a prior company I once tried appending to the list of NFS exports, but dropped the "no-root-squash" option, and instantly denied write permissions to our entire VMware farm. You can imagine what then happened to all of the VMs for this mission critical customer. =P
I wouldn't want to be the person who wrote the wrong command! Sheesh.
I brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Sounds like an opportunity for machine learning. Anyone want to write an AI BOFH?
I'm surprised how transparent this is, I can find Amazon often a bit opaque when dealing with issues.