421–430 of 535 posts
Oh, that interview question. “Tell me about something you broke in your last job"
I did all this while sitting 2 feet from a print out of "The 8 fallacies of distributed systems". Bandwidth is indeed not infinite, can confirm.
Earlier quoted context omitted.
Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".
Agreed, especially regarding the culture but isn't this pretty much the same explanation they gave a few years ago when something similar happened? I seem to recall an EC2 or S3 outage a few years ago that boiled down to an engineer pushing out a patch that broke an entire region when it was supposed to be a phased deployment. I could be mis-remembering that but it's important that these lessons be applied across the…
(Source: am a self-identified post-mortems connoisseur. :)
Earlier quoted context omitted.
To make error is human. To propagate error to all server in automatic way is #devops - DevOps Borat
I've long said something like "To err is human. To fuck up a million times in a second you need a computer." I may have to upgrade that to take the mighty power of Cloud (TM) into account, though. Billions and trillions of fuck ups per second are now well within reach!
This quote (paraphrased) actually dates all the way back to 1969:
> To err is human; to really foul things up requires a computer.
Earlier quoted context omitted.
I can't wait until quantum computing lets us add a degree of simultaneity to fucking up. Fuck up in many ways... AT ONCE!
It will not be certain if you have fucked up or not until you actually go to check.
Earlier quoted context omitted.
What bothered me about running TrueTech is that customers would sometimes demand repercussions against employees for making mistakes. Enter Frans Plugge. Whenever a customer would get into that mode we'd fire Frans. This was easy, simply because he didn't exist in the first place (his name was pulled from a skit by two Dutch comedians, bonus points if you know who and which skit). This usually then caused the custome…
It is unreasonable for me to think that company owners should have the spine to say, "We take the decision to fire someone very seriously. We'll take your comments under consideration, but we retain sole discretion over such decisions." It irks me that businesses fire people because of pressure from clients or social media. But having never been the boss, I may be missing something.
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Yeah as soon as I read this I felt bad for the employee. I remember writing an update statement without a where clause and having to restore the table from backup. But that was at a company not as advanced as Amazon. Fat fingering a key like that is just crazy (but comforting that even at Amazon it happens) and I'm sure they've fixed that from happening again.
Earlier quoted context omitted.
Fascinating! Do you know of any source that documents the claim that Robert Allen made this statement?
I'll try. It was a dark day. We all watched the news and I recall him saying that. The specific quote I don't remember but it was something like " you can consider that I did." I think he was asked what will happen to the person that caused it and who is that person. Everyone knew right away this had to be human error. Right away. Switches simply had too much redundancy. It was big then and not sure if I can locate a…
> As far as our customers are concerned, I did it.
http://www.upi.com/Archives/1990/01/16/ATT-pinpoints-cause-o...
This part is also interesting: > While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post: > We build our systems with the assu…
> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…
Earlier quoted context omitted.
This is true in some cases, but not when mitigations aren't practiced properly - it's not the fat fingered user who should be fired or retrained, but the designer or maintainer of the system that allowed it to become a serious issue. Look at the recent GitLab incident - one guy messed up and nuked a server. Okay, that happens sometimes, go to backups. Uh oh, all the backups are broken. Minor momentary problem just tu…
Fair enough. I guess what I meant was specifically using training or punishment to combat "momentary lapse" issues. If someone doesn't test their backups, you train them to test backups. If someone lies about testing the backups, maybe you fire them. But if someone trips and shatters the only backup disk, you don't yell at them - you create backups that an instant of clumsiness can't ruin. I did overstate, training i…
Instead, you make a machine verify the backups simply by using the backups all the time. For example, at work I feed part of our data pipeline with backups: Those processes have no access to the live data. If the backups break, those processes would provide bad information to the users, and people would come complaining in a matter of minutes.
Just like when you have a set of backup servers, you don't leave them collecting dust, or tell someone to go look at them every once in a while: you just route 1% of the traffic through them. They are still extra capacity, you can still do all kinds of things to them without too much trouble, but you know they are always working.
Never, ever, force people to do things they don't gain anything from. Their discipline would fade, just like it fades when you force them to a project management tool they get no value from.