Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

81–90 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#81

Post by schneems »

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

I see this as a problem of not investing enough in the deploy process. (Disclosure: I maintain an open source deploy tool for a living).

Charity Majors gave a talk in Euruko that talked a lot about this. Deploy tooling shouldn’t be a bunch of bash scripts in a trench coat, it should be fully staffed, fully tested, and automated within an inch of its life.

If you have a deploy process that has some king of immutable architecture, tooling to monitor (failed/stuck/incomplete) rollouts, and the ability to quickly rollback to a prior known good stage then you have layers of protection and an easy course of action for when things do go sideways. It might not have made this problem impossible, but it would have made it harder to happen.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#82

Post by m463 »

daft_pink wrote: post #39

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

> It's like writing code that flies a jumbo jet.

and upgrading it from a coach seat

Re: Knightmare: A DevOps Cautionary Tale (2014)

#83

Post by justinclift »

lwhi wrote: post #75
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

Automated deployments require planning before the time they're executed. If code is involved, someone likely reviews and approves it. There are naturally far more safeguards in place than there would be for a manual deployment.

In an ideal world, sure.

In the current one, we have Facebook's "Move fast and break things" being applied to many things where it has no business being.

Banking and communications infrastructure comes to my mind, but there are definitely others. :)

Re: Knightmare: A DevOps Cautionary Tale (2014)

#84

Post by Waterluvian »

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

Which really means is a failure of leadership for being so incompetent as to allow such an intensely risky situation to exist.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#85

Post by jonplackett »

xyst wrote: post #76

Having worked in some Fortune 500 financial firms and low rent “fintech” upstarts, I am not surprised this happened. Decades of bandaid fixes, years of rotating out different consultants/contractors, and software rot. Plus years of emphasizing mid level management over software quality. As other have mentioned, I don’t think “automation of deployment” would have prevented this company’s inevitable downfall. If it was…

The thing I was surprised about is that they survived!

After all that they still got a $400 million cash bailout!

Re: Knightmare: A DevOps Cautionary Tale (2014)

#86

Post by yardstick »

coldtea wrote: post #49
pavas wrote: post #42

My team's systems play a critical role for several $100M of sales per day, such that if our systems go down for long enough, these sales will be lost. Long enough means at least several hours and in this time frame we can get things back to a good state, often without much external impact. We too have manual processes in place, but for any manual process we document the rollback steps (before starting) and monitor th…

> My team's systems play a critical role for several $100M of sales per day, such that if our systems go down for long enough, these sales will be lost. Would they? Or would they just happen later? In a lot of cases in regular commerce, or even B2B, the same sales can often be attempted again by the client for a little later, it's not "now or never". As a user I have retried things I wanted to buy when a vendor was d…

It would be a serious issue for in person transactions like shops, supermarkets, gas stations, etc

Imagine Walmart or Costco or Chevron centralised payment services went down for 30+ mins. You would get a lot of lost sales from those who don’t carry enough cash to cover it otherwise. Maybe a retailer might have a zapzap machine but lots of cards aren’t imprinted these days so that’s a non starter too.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#87

Post by hooverd »

schneems wrote: post #81
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

I see this as a problem of not investing enough in the deploy process. (Disclosure: I maintain an open source deploy tool for a living). Charity Majors gave a talk in Euruko that talked a lot about this. Deploy tooling shouldn’t be a bunch of bash scripts in a trench coat, it should be fully staffed, fully tested, and automated within an inch of its life. If you have a deploy process that has some king of immutable a…

> bash scripts in a trench coat That's an amazing turn of phrase.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#88

Post by yieldcrv »

goldinfra wrote: post #78
envsubst wrote: post #66

Earlier quoted context omitted.

[flagged]

Most jobs do in fact contribute to the well being of humanity, however little. It's few jobs, like most in financial trading, that actively reduce the well being of humanity. Never will you meet a more self-deluded and pathetic set of humans. Desperate money addicts that often become other kinds of addicts. Whole thing should be abolished. Source: I worked in finance when I was young and dumb.

I, for one, am so glad to “own” the “compose” button at a democracy destabilizing adtech-conglomerate

Re: Knightmare: A DevOps Cautionary Tale (2014)

#90

Post by Waterluvian »

daft_pink wrote: post #39

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

It’s not scary when it’s done properly. And done properly can look like an incredibly tedious job. I think it’s for a certain kind of person who loves the process and the tests and the simulators and the redundancy. Where only 1% of the engineering is the code that flies the plane.
Post reply on HN