One of the largest AWS outages to date was caused by a scenario like this. [1] A mistyped commanded removed too many servers from an S3 subsystem, overloading the remaining servers and crashing the subsystem. The failure snowballed until the entire S3 region was down, which then caused issues with dependent services like EBS, ALB, and Lambda. They couldn't even update the status page because that also depended on S3.
Type in the exact number of machines to proceed
81–90 of 340 posts
Re: Type in the exact number of machines to proceed
#82I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling
> I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. The concept makes sense, though I don't quite fully get how to translate it to other contexts besides train driving where unexpected and unpredictable events come up all the time. Let's say you're driving a car and the traffic light turns red. D…
Let's say you get a request to delete users #s 1, 17, 152, and 43.
Now you can have the request and database administration tool open and point and call at the numbers and any queries and make sure you are deleting the right users.
Re: Type in the exact number of machines to proceed
#83Earlier quoted context omitted.
Even having a dry run mode is exciting. Doesn't even have to give complete results just "I was planning to delete 3 files and create 7 files", gives a hint whether the command will blow up the system or not.
I wish SQL had a dry-run mode in updates and deletes for that reason. "Run it as a query first" gets 90% of the way until you drop a constraint by accident whilst rewriting it as an update :o
[deleted]
Re: Type in the exact number of machines to proceed
#84Earlier quoted context omitted.
Even having a dry run mode is exciting. Doesn't even have to give complete results just "I was planning to delete 3 files and create 7 files", gives a hint whether the command will blow up the system or not.
I wish SQL had a dry-run mode in updates and deletes for that reason. "Run it as a query first" gets 90% of the way until you drop a constraint by accident whilst rewriting it as an update :o
Transactions and rollback is the dry run. The problem is that if you keep the transaction open for too long, you will block other updates to the same data.
Re: Type in the exact number of machines to proceed
#85I have a habit of creating cli tools, which potentially do dangerous things, to default to dry-run mode. For example, instead of the typical `--dry-run` or `-n` option, my scripts instead had a cheesy `--do-it` to be non-dry-run. It is annoying as hell to my colleagues, but saved the day many times.
works great until some asshole puts
alias harikb_script='harikb_script --do-it'
Re: Type in the exact number of machines to proceed
#86One of the largest AWS outages to date was caused by a scenario like this. [1] A mistyped commanded removed too many servers from an S3 subsystem, overloading the remaining servers and crashing the subsystem. The failure snowballed until the entire S3 region was down, which then caused issues with dependent services like EBS, ALB, and Lambda. They couldn't even update the status page because that also depended on S3.…
I remember that. The AWS dashboard was all green checkmarks... because the red checkmarks icons the dashboard was supposed to display were stored inside the crashed servers.
Re: Type in the exact number of machines to proceed
#87Earlier quoted context omitted.
I wish SQL had a dry-run mode in updates and deletes for that reason. "Run it as a query first" gets 90% of the way until you drop a constraint by accident whilst rewriting it as an update :o
Transactions and rollback is the dry run. The problem is that if you keep the transaction open for too long, you will block other updates to the same data.
Yep, I always write any update queries as a rollback transaction with some selects inside it to verify what the data looks like after it's done now, before I switch it to commit. I primarily use Microsoft SQL Server right now, so I also use WITH (NOLOCK) to prevent issues running my query will have with other updates.
Re: Type in the exact number of machines to proceed
#88Side question. How many/which companies have more than one million Linux machines?
The number blew me away. But does she mean in one location or VMs?
One million is a lot no matter how you slice it.
Re: Type in the exact number of machines to proceed
#89Reminds me of the proposal to keep the nuclear launch codes inside the body of an innocent volunteer, so the President would have to kill the person to get the codes. https://boingboing.net/2015/12/11/proposal-keep-the-nuclear-...
I've never understood this idea. If you believe we should never use nuclear weapons, then don't have them at all. If you believe there is a case where it may be moral and rational to use nuclear weapons, why would you want to put a potential barrier in the way of their use? You could have a situation where everyone was agreed to use them but the president was physically unable to harm the aide to use them. You can kn…
You put too much confidence in human reason.
Everybody agrees that this is a nuke-them-all situation, but the president, given himself part of the task of ripping apart human bodies, thinks more about the subject and decides a another diplomatic round is a better option.
Re: Type in the exact number of machines to proceed
#90I have a habit of creating cli tools, which potentially do dangerous things, to default to dry-run mode. For example, instead of the typical `--dry-run` or `-n` option, my scripts instead had a cheesy `--do-it` to be non-dry-run. It is annoying as hell to my colleagues, but saved the day many times.
I do something similar with my scripts, but have `--go` action, even on a script that requires no other options, just so that if it's run without any options, the person running it gets a message saying what the script WOULD do, if `--go` were passed in.