331–340 of 535 posts
This is the risk you run into by doing everything through the command line. This would be really hard to do through a good GUI.
I highly doubt this claim, humans make mistakes regardless of the control method. In this particular case the scripts didn't have adequate protections in place, but that's the benefit of hindsight
This is the risk you run into by doing everything through the command line. This would be really hard to do through a good GUI.
It would be equally as hard through a good command line. Why is good design only possible in GUIs?
A good command line would better protection would have helped, but this fails some of the core of good interface design.
Earlier quoted context omitted.
Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.
FWIW: Setting "safe-updates=1" in ~/.my.cnf will require UPDATE and DELETE statements in the client to have a WHERE clause which references a key. It's not perfect protection, but it will save you from a lot of mistakes.
My worst DELETE fail however was:
DELETE * FROM table WHERE [long condition that resolves to true for all records]
It's not my personal habit but some folks I know turn off auto commit and BEGIN a transaction every time they enter an interactive SQL sessions. They then default to ROLLBACK at least once before COMMITing them.
That and having a user with read-only permissions or a read replica
Earlier quoted context omitted.
Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".
Which is really the point of automation and configuration management. When a manager asks you, "How are you going to prevent this in the future?" You can say, "We added a check so n must be less than x% of the total number of cluster members," or "We added additional unit tests for the missing area of coverage" or "We added new integration tests that will pick up on this." Tests and configuration scripts don't preven…
By the way - this is not just Amazon's problem now. We know the internet has a single point of failure. So does a lot of IoT.
When will we experience the first Suicide DevOps?
Earlier quoted context omitted.
I think it comes down to how important your ELB logs are -- if they are important enough that you don't want to allow traffic without logs (i.e. if you're using them for some sort of auditing/compliance), then failing when it can't write the logs seems like the right choice.
Thanks, that is a fair perspective. In our case we're using ELB logs as a redundant trace and it isn't critical that our traffic stops if the access logs fail. It would be nice if this behavior became a toggle in ELB settings, but think we can set something up to disable access logs programatically if we start seeing S3 issues.
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
To be fair, the real problem isn't that someone screwed up a playbook or command. The real problem is that a tiny mistake in a command can cause an entire service to be disrupted for hours. That's the problem that needs to be fixed.
Earlier quoted context omitted.
Yeah indeed. You know who the one person at Amazon is that I'd expect to never fat finger a sensitive command ever ever again? The guy who managed to fat finger S3 on Tuesday. Firing him over this mistake is worse than pointless, it offers absolution to every other developer and system that helped cause this event.
That old chestnut. Is it true?
Take a moment to look at the construction of this report. There is no easily readable timeline. It is not discoverable from anywhere outside of social media or directly searching for it. As far as I know, customers were not emailed about this - I certainly wasn't. You're an important business, AWS. Burying outage retrospectives and live service health data is what I expect from a much smaller shop, not the leader in…
Also notably missing is the "we will automatically refund all affected customers" line that we'd expect from somebody who wants to provide excellent service. A graphical illustration of the service dependencies they were talking about would have been nice as well.
I wouldn't want to be the person who wrote the wrong command! Sheesh.
I brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.
Fixed that by putting the DC domain in red on the prompt.