Wow, I'm sure that person who mis entered the command will never, ever, ever, do it again.
Summary of the Amazon S3 Service Disruption
531–535 of 535 posts
Re: Summary of the Amazon S3 Service Disruption
#532Earlier quoted context omitted.
Any time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.
What bothered me about running TrueTech is that customers would sometimes demand repercussions against employees for making mistakes. Enter Frans Plugge. Whenever a customer would get into that mode we'd fire Frans. This was easy, simply because he didn't exist in the first place (his name was pulled from a skit by two Dutch comedians, bonus points if you know who and which skit). This usually then caused the custome…
To what lengths did you keep that up? Did you just tell the client that informally, formally in a reunion or so, or did you actually put it in writing or even fake some "firing" paperwork?
Re: Summary of the Amazon S3 Service Disruption
#533Earlier quoted context omitted.
You don't have to know what would be a mistake. E.g. if the tool is used most of the time to operate on a small set of servers, you have some extra confirmation or command-line option for removing a large set. That's good UI design in tools with powerful destructive capabilities. You make the UI to do lots of things v.s. the few things you do routinely different enough that there's no mistaking them.
You can also have the program tell the user what's going to happen (if it can be computed beforehand), e.g. "This will affect 138 server(s)."
This is a good pattern to use. The more pre-feedback I get, the less likely I am to make a horrible mistake.
However one problem I often see with this pattern is the numbers are not formatted for humans to read. Suppose it prompts:
"1382345166 agents will be affected. Proceed? (y/n)"
Now if the number is formatted for a human to consume, I don't have to break flow and am much less likely to make an "order-of-magnitude error":
"1,382,345,166 (1.4M) agents will be affected. Proceed? (y/n)"
Re: Summary of the Amazon S3 Service Disruption
#534Earlier quoted context omitted.
You can also have the program tell the user what's going to happen (if it can be computed beforehand), e.g. "This will affect 138 server(s)."
This doesn't work. Users learn to ignore the message.
I think it depends on the quality of the feedback. Most tooling sucks, so the messages are very literal trace statements peppered through the code. , vs what the user-facing impact will be. When the thing is just spitting raw information at me, I'm probably going to train myself to ignore it. But if it can tell me what is going to happen, in terms that I care about, then I'll pay attention.
Imagine I just entered a command to remove too many servers that will cause an outage:
"Finished removing servers"
(better than no message, I suppose)
"Finished removing 8 servers"
(better, it's still too late to prevent my mistake
but at least I can figure out the scale of my mistake)
"8 servers will be removed. Press `y` to continue"
(better, no indication of impact but if I'm paying
attention I might catch the mistake)
"40% capacity (8 servers) will be removed.
Load will increase by 66% on the remaining 12 servers.
This is above the safety threshold of a 20% increase.
You can override by entering `live dangerously`."
(preemptive safety check--imagine the text is also red so it stands out)Re: Summary of the Amazon S3 Service Disruption
#535Earlier quoted context omitted.
We have a deploy script that does exactly this, unfortunately we've all gotten our muscle memory so trained that most of us type the deploy command, press enter, type yes and press enter before we're ever even prompted. Fortunately in most cases a quick ctrl-c can prevent any actual damage.
Type 'yes'? Make it ask you to type out the deploy location. I think that helps a lot.
I think the only way to nuke the muscle memory from this equation would be to have it make one type a random dictionary word (or solve an arithmetic problem or something - that one might help prevent drunk deploys ;).
If the prompt was type "production" to confirm, I'm sure I'd just as readily train myself to jump the gun on that one.