Knightmare: A DevOps Cautionary Tale (2014) • Hacker News Board

Re: Knightmare: A DevOps Cautionary Tale (2014)

#31

Post by erinaceousjones » Thu, Feb 06, 2020, 10:51 AM UTC

lostlogin wrote: post #5

floatingatoll wrote: post #3

Previous discussions on HN: 2014: https://news.ycombinator.com/item?id=7652036 2015: https://news.ycombinator.com/item?id=8994701

Thanks - the top comment from vijucat in the 2015 discussion is anxiety inducing. “ - Ctrl-r for reverse-search through history - typing 'ps' to find the process status utility (of course) - pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh)”

I had a habit of doing `sudo shutdown now` on my desktop as I'm leaving my office. I don't know why, it takes longer than simply hitting the power button.

Didn't notice I was still SSH'ed into "the" server which was at the time a single point of failure for my entire project, and as a lowly not-an-IT-person-just-a-developer in our corporate environment, I didn't have access to the machine to go power it back on. And the IT people I knew who could help had gone home for the day.

Felt super dumb writing that up in the downtime log the next day.

Having read this article, it makes me super glad I'm working on very niche slow-paced stuff which, when goes down for ~12 hours, is a minor annoyance to our users rather than "you're costing us millions of $currency per minute" :-)

Re: Knightmare: A DevOps Cautionary Tale (2014)

#32

Post by HelloNurse » Thu, Feb 06, 2020, 11:44 AM UTC

bobbiechen wrote: post #12

SteveNuts wrote: post #11

Earlier quoted context omitted.

A lot of times those issues have been "fully automated (but with human inputs)" or "fully automated with no guardrails"

This seems to cover all the cases. Either there are guardrails (as human inputs), or there aren't. Unless I'm missing a middle ground here?

Automated checks. For example, in this case, confirming that the other containers are quiescent (as they are supposed to be) and locking them down before the potentially conflicting operation.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#33

Post by swish_bob » Thu, Feb 06, 2020, 1:06 PM UTC

evilotto wrote: post #30

toolslive wrote: post #29

off topic, but a "knightmare" is also a chess term. It's a good-knight-vs-bad-bishop position that went horribly wrong for the owner of the bishop.

Also a Batman alternate timeline

And a late 80s early 90s children's TV programme.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#34

Post by bobbiechen » Thu, Feb 06, 2020, 8:35 PM UTC

HelloNurse wrote: post #32

bobbiechen wrote: post #12

Earlier quoted context omitted.

This seems to cover all the cases. Either there are guardrails (as human inputs), or there aren't. Unless I'm missing a middle ground here?

Automated checks. For example, in this case, confirming that the other containers are quiescent (as they are supposed to be) and locking them down before the potentially conflicting operation.

I'd be willing to bet that it's extremely rare for a fully automated process to have absolutely no guardrails/checks/tests, and it's also extremely rare for a fully automated process to have 100% test coverage.

If this check existed and the system failed in some other way, it would be characterized as "fully automated with no guardrails" (for the scenario which caused the failure). "We had tests but missed an edge case" usually doesn't get you any sympathy.

So what's left? Formally proving correctness is overkill for most things. The "end-to-end" argument [1] might be able to detect when something goes wrong at the end to rollback or alert, but what if the intermediate steps have already caused damage or prevent the "end" from being reached entirely? If a run is taking longer than usual, how do you differentiate between harmless delays in the intermediate steps, and the run being entirely broken somehow?

[1] http://pages.cs.wisc.edu/~bart/739/papers/end-to-end.pdf