Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type in the exact number of machines to proceed

rachelbythebay.com

Post reply on HN
Print view View original

151–160 of 340 posts

Re: Type in the exact number of machines to proceed

#151

Post by crispyambulance »

bronco21016 wrote: post #120

It amazes me that something like this can be done by a single person. In aviation any time input is given to the machine, it's entered by one human (typically pilot flying) and then verified by the other human (typically pilot monitoring) before being committed to or executed. For example... when a new altitude is assigned by ATC, say FL300, the pilot flying will spin it in the selector window and keep his hand or fi…

> It amazes me that something like this can be done by a single person.

In many dysfunctional orgs, having someone to blame is desirable. They will use all kinds of words for it like "accountability".

But at the end of the day, heros who take stupid risks that succeed get rewarded, cautious people that ask questions and try to understand before acting are smugly dismissed, and would-be heroes that burn the house down because of recklessness get blamed and make everyone else look good. It's all too common.

Re: Type in the exact number of machines to proceed

#153

Post by Darkphibre »

Reminds me of when the Fortune 50 company (150k employees) I worked for rolled out new firewall restrictions that blocked the DNS port.

To all machines. Employee and servers alike.

Yes. Including the DNS servers.

Took them a day or two to work out how to roll that one back.

Re: Type in the exact number of machines to proceed

#154

Post by scottlamb »

rossjudson wrote: post #145

This resonates with me. Years ago I took down a service in a cell accidentally (Googlers might empathize: never 'borg' when you meant to 'borgcfg'). If I had been asked to enter the exact number of tasks I was about to nuke, I might have thought twice ;)

I've certainly deliberately downed an enormous number of tasks, though, as part of a cluster turn-down. I love the technique of requiring the operator to echo a key fact, but in the case you're describing I think the key fact is not how many tasks but that that they're serving live traffic. So:

* You could ask the operator to echo the qps figure...but really any number other than zero is likely to be an error, so it can just error out in that case without needing the confirmation.

* Even if it is serving zero qps now, if it's not explicitly drained at the load balancer, downing it is likely to be a mistake. So even better to check that.

Only once in my career have I taken down jobs serving live traffic. (They were serving 100% errors.) It was deliberate, but even so I wouldn't have minded having to supply a --yes-i-know-im-downing-live-jobs.

edit: and if for some reason my assumption is wrong and downing undrained things becomes routine...well, you'd want to fix that, but as a short term measure going back to the confirming a number rather than the force option would be appropriate. Is certainly not good to have an override that's routinely used.

Re: Type in the exact number of machines to proceed

#155

Post by mehrdadn »

brundolf wrote: post #127
csmattryder wrote: post #8

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling

I've only been in the job field for six years, and yet: My first boss accidentally deleted our QA database, meaning to delete a local copy A later boss accidentally deleted our production database, thinking it was the clone that he had just made (which luckily we still had) Both of them were very experienced developers in their 40s. Nobody is beyond this kind of mistake.

Reminds me of when I accidentally deleted a virtual hard disk I had a few years ago, because I'd copied it earlier and I thought I still had the other copy left. Only afterward did I remember I'd done the exact same thing to the other copy earlier... thankfully the information on it wasn't critical, but it was kind of terrifying to realize it very well could have been.

Re: Type in the exact number of machines to proceed

#156

Post by xaedes »

_ikke_ wrote: post #124
xaedes wrote: post #95

Earlier quoted context omitted.

This would help a bit: Don't accept the "--do-it" as first parameter, make it obligatory to be the last.

my_command() { command my_command "$@" --do-it }

Good point. Stuff like this is why I wrote "a bit". Thank you for providing an example, why it wont be enough.

Re: Type in the exact number of machines to proceed

#157

Post by cbm-vic-20 »

dingaling wrote: post #76
jrumbut wrote: post #72

Earlier quoted context omitted.

Even having a dry run mode is exciting. Doesn't even have to give complete results just "I was planning to delete 3 files and create 7 files", gives a hint whether the command will blow up the system or not.

I wish SQL had a dry-run mode in updates and deletes for that reason. "Run it as a query first" gets 90% of the way until you drop a constraint by accident whilst rewriting it as an update :o

MySQL has a command line option "--i-am-a-dummy" (aka "--safe-updates") for exactly this purpose.

https://dev.mysql.com/doc/refman/8.0/en/mysql-command-option...

Re: Type in the exact number of machines to proceed

#158

Post by skymt »

dingaling wrote: post #76
jrumbut wrote: post #72

Earlier quoted context omitted.

Even having a dry run mode is exciting. Doesn't even have to give complete results just "I was planning to delete 3 files and create 7 files", gives a hint whether the command will blow up the system or not.

I wish SQL had a dry-run mode in updates and deletes for that reason. "Run it as a query first" gets 90% of the way until you drop a constraint by accident whilst rewriting it as an update :o

Enough folks have replied that transactions are the way to go, but I just wanted to add that whatever interface tool you use for your database may have an option to force you to commit your transactions manually. For example PostgreSQL's default 'psql' shell has the "autocommit" option which, when disabled, requires you to manually 'commit;' before any changes take effect.

Re: Type in the exact number of machines to proceed

#159

Post by mehrdadn »

greedo wrote: post #149
Gene_Parmesan wrote: post #130

Earlier quoted context omitted.

Firing the person who happened to be at the wheel when a mistake like this occurs never seems like the right choice to me, especially if their performance to-date had otherwise been good. Everybody has off days, or just instances where circumstances misalign in just the wrong way. To pretend otherwise is silly; instead, it's the leader's/team's responsibility to ensure that those sort of off days don't lead to massiv…

Sometimes. And sometimes they make the same mistake over and over. We had an admin in charge of our storage. He had worked with our old vendor's SAN for years, then we got a new SAN. Trained him/certified him etc. He "accidentally" shut down the entire SAN. That brought down the entire company for over 9 hours. Fast forward two years later, he screwed up again and caused a storage outage affecting about 1100 VMs. Luc…

3 mistakes in... >2 years? I feel like it's really hard to tell if the problem is really the person at that point. Have you had others perform the same job for a similar duration to see if they avoid the same mistakes?

Re: Type in the exact number of machines to proceed

#160

Post by greedo »

mehrdadn wrote: post #159
greedo wrote: post #149

Earlier quoted context omitted.

Sometimes. And sometimes they make the same mistake over and over. We had an admin in charge of our storage. He had worked with our old vendor's SAN for years, then we got a new SAN. Trained him/certified him etc. He "accidentally" shut down the entire SAN. That brought down the entire company for over 9 hours. Fast forward two years later, he screwed up again and caused a storage outage affecting about 1100 VMs. Luc…

3 mistakes in... >2 years? I feel like it's really hard to tell if the problem is really the person at that point. Have you had others perform the same job for a similar duration to see if they avoid the same mistakes?

This is just the mistakes made in the SAN/Storage part of his responsibilities. As we used to say in World of Warcraft, "Can't heal stupid."
Post reply on HN