Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type in the exact number of machines to proceed

rachelbythebay.com

Post reply on HN
Print view View original

321–330 of 340 posts

Re: Type in the exact number of machines to proceed

#321

Post by greedo »

jodrellblank wrote: post #231
greedo wrote: post #149

Earlier quoted context omitted.

Sometimes. And sometimes they make the same mistake over and over. We had an admin in charge of our storage. He had worked with our old vendor's SAN for years, then we got a new SAN. Trained him/certified him etc. He "accidentally" shut down the entire SAN. That brought down the entire company for over 9 hours. Fast forward two years later, he screwed up again and caused a storage outage affecting about 1100 VMs. Luc…

> " He had worked with our old vendor's SAN for years, then we got a new SAN. " Great way to invalidate years of experience. Presumably from your telling of the story, he didn't cause problems with the old vendor's SAN? > " He "accidentally" shut down the entire SAN. " So, was it an accident, or was it an "accident"? You can't have it being a mistake if you're also hinting it was deliberate and malicious.

He was trained and certified on the new SAN, and surely some of his prior experience on the legacy SAN would translate. Just as moving from AIX to RHEL/CentOS wouldn't invalidate all your skills and experience.

It was a real accident when he shut down the SAN the first time. I don't know why I put it in scare quotes.

Re: Type in the exact number of machines to proceed

#322

Post by im3w1l »

rachelbythebay wrote: post #126
bronco21016 wrote: post #120

It amazes me that something like this can be done by a single person. In aviation any time input is given to the machine, it's entered by one human (typically pilot flying) and then verified by the other human (typically pilot monitoring) before being committed to or executed. For example... when a new altitude is assigned by ATC, say FL300, the pilot flying will spin it in the selector window and keep his hand or fi…

Best practice for using the "weaponized" version of the tool when you had powers to actually hit all of them at once was to paste the command into IRC and get some of your fellow peeps to eyeball it and make sure it was sane. team: hey, sanity check this please: hsh -A "dumb_thing && other_thing --foo --bar" shipit [ I type the command ] ok, running as job 1234 The last part was a courtesy done so that they could wat…

I think an automated tool would be preferable since there is no 100% foolproof guarantee that what you type in irc is the same as what you type in the terminal.

Re: Type in the exact number of machines to proceed

#323

Post by beefield »

yencabulator wrote: post #318
beefield wrote: post #308

Earlier quoted context omitted.

If someone just gave me a tool to show me the expected wall time of query before actually running it, I would be quite happy. I would not even need that much of accuracy, anything up to one order of magnitude would be useful, and even up to two orders of magnitude I would use occasionally.

https://www.postgresql.org/docs/current/using-explain.html ?

Nobody has ever been able to give me a function query cost -> wall time with any accuracy.

Re: Type in the exact number of machines to proceed

#324

Post by jeffbee »

scottlamb wrote: post #319
jeffbee wrote: post #255

Earlier quoted context omitted.

The way we approached this on my SRE team was semi-manual with improved ergonomics. We embedded the live traffic graph in the turndown tool, so it would be right in your face before you took the destructive action. Of course it was always possible to go one level down on the tooling and do everything manually, but it wasn't the usual way.

Seems reasonable, but as you might have seen, rossjudson did accidentally-ish go to a lower layer: he wrote "never 'borg' when you meant to 'borgcfg'". And you're still relying on someone actually looking at the graph in their face which isn't as sure a thing as it'd be if they had to echo something back as Rachel is advocating for. (For the benefit of non-Googlers/Xooglers: borg is a lower-level tool mostly used whe…

I really agree with your philosophy here but I've never been able to perfect it in practice. The imperfection comes from the way there is inevitably some mapping of things to other things by name. I can ask a load balancer whether clients of a service are being sent to a named capacity or not (i.e. is the thing I want to remove "drained") but that doesn't rule out the possibility that another service maps a different name to the same backend and I forgot to integrate that name with my automation. Also impossible to rule out that a client exists which bypasses or ignores the advice of the load balancer. Having visibility into caller identity helps a lot with this kind of problem but outside of Google there is a scary word called "cardinality" which prevents people from monitoring the whole caller×server space.

Re: Type in the exact number of machines to proceed

#325

Post by scottlamb »

jeffbee wrote: post #324
scottlamb wrote: post #319

Earlier quoted context omitted.

Seems reasonable, but as you might have seen, rossjudson did accidentally-ish go to a lower layer: he wrote "never 'borg' when you meant to 'borgcfg'". And you're still relying on someone actually looking at the graph in their face which isn't as sure a thing as it'd be if they had to echo something back as Rachel is advocating for. (For the benefit of non-Googlers/Xooglers: borg is a lower-level tool mostly used whe…

I really agree with your philosophy here but I've never been able to perfect it in practice. The imperfection comes from the way there is inevitably some mapping of things to other things by name. I can ask a load balancer whether clients of a service are being sent to a named capacity or not (i.e. is the thing I want to remove "drained") but that doesn't rule out the possibility that another service maps a different…

I agree you can never reach perfection. I expect there'd still be postmortems with "Our safety check was missing/bad" in the "what went wrong" section for various project-specific technical reasons. But I'd expect there to be (a) fewer such postmortems, and (b) an action item to fix the job's safety check service specification and audit the team's other ones, rather than the rather inexcusable IMHO "this tool doesn't support those, /shruggie, maybe schedule more training about which tool to use".

Re: Type in the exact number of machines to proceed

#326

Post by rpeden »

pantalaimon wrote: post #316
ajb wrote: post #294

Earlier quoted context omitted.

Not the op, but non stick pans will burn if the pan is heated while empty.

I think non-stick pans are a fad. A well greased iron or steel pan works much better and is impossible to destroy.

Can non-stick pans even be a fad when Teflon coated cookware has been popular for 60+ years?

Re: Type in the exact number of machines to proceed

#327

Post by rkagerer »

morelisp wrote: post #178
js2 wrote: post #99

Earlier quoted context omitted.

I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter. I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells. [0] Tec…

A similar tip I picked up long ago: If you're typing a dangerous command, first type a `#` (or `--` if it's SQL, etc.), then the command. Then read it. Then go back to the start of the line and remove the comment and run it.

Especially useful if you're remoted in over a laggy connection.

Re: Type in the exact number of machines to proceed

#328

Post by efreak »

GauntletWizard wrote: post #306
PureParadigm wrote: post #240

Earlier quoted context omitted.

A coworker of mine would write all his bash scripts to echo out the commands it would run, and then to actually run it he would pipe it to bash. This way he could inspect the commands to make sure they were correct before running them. Something like: ./dangerous-script.sh $args | bash

I would love a shell that allows you to “run” a script in manual mode - Where at the end of every command, every statement, it prints what the next command will be with all variables expanded or otherwise called out, and then requires you to hit “enter” to cause it to proceed. I write a decent amount something between README and Shell Script. I’ve already got an awk one-liner that parses the shell out of Markdown. I…

Press f8 to process autoexec?

Re: Type in the exact number of machines to proceed

#329

Post by nucleardog »

mcintyre1994 wrote: post #286

AWS sometimes does something similar to this like “enter the name of the thing you’re trying to delete to confirm”. I think it makes sense because you can have such a huge difference between how much you care about certain s3 buckets or CloudFormation deploys etc. In true AWS fashion it’s inconsistent between services though.

To their credit, even if it’s unintentional, every time one of those screens pop up I have to stop and think about what I’m doing because every screen wants something different from me!

Re: Type in the exact number of machines to proceed

#330

Post by gknoy »

brundolf wrote: post #192
MaxBarraclough wrote: post #190

Earlier quoted context omitted.

Solid tip. For GUI-enabled servers, use distinctively coloured wallpapers. I recommend bright red for production machines. The image itself can be just about anything, provided the colour is clear. Doesn't hurt to use an image that's related to the server's purpose, and to put the name of the server right there in the wallpaper somewhere.

That's nifty, but sounds like more effort than changing a single color in one's .bash_profile

Using iterm2, you can set a "badge" (large text overlay) on a terminal tab. I have a short shell function (`ib foo`) that sets the badge to arbitrary text. It's NOT as good as setting the terminal theme, but it's still very helpful to use it like this:

    ib production && ssh production-machine
    ib demo && ssh demo-machine
It's definitely helped me when testing a fix on a demo or staging instance, and has helped me avoid doing it on production accidentally.
Post reply on HN