Knightmare: A DevOps Cautionary Tale (2014) • Hacker News Board

Re: Knightmare: A DevOps Cautionary Tale (2014)

#61

Post by moeris » Sun, Sep 10, 2023, 9:15 PM UTC

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

Automated deployments would have allowed you to review the deployment before it happened. A failed deployment could be configured to allow automatic rollbacks. Automated deployments should also handle experiment flags, which could have been toggled to reduce impact. There are a bunch of places where it could have intervened and mitigated/prevented this whole situation.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#62

Post by stusmall » Sun, Sep 10, 2023, 9:15 PM UTC

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

I think the big improvement would be consistency. Either all servers would be correct or all servers would be incorrect. The step where "Since they were unable to determine what was causing the erroneous orders they reacted by uninstalling the new code from the servers it was deployed to correctly" wouldn't have had a negative impact. They could have even instantly rolled back. Also if they were using the same automated deployment processes for their test environment they might have even caught this in QA.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#63

Post by siftrics » Sun, Sep 10, 2023, 9:16 PM UTC

UncleMeat wrote: post #34

Nevermark wrote: post #23

Earlier quoted context omitted.

Leeching implies someone has found a way to skim value from you without providing value. Someone taking on loads of risk to carry out your commands efficiently is providing value. You can argue whether they are doing so competently or not, or whether they are pricing optimally or not, but they are not just ”takers” or “leeches”.

How are they taking on loads of risk? Risk has a particular meaning in investment and "well, a bug can blow up my company" isn't part of that meaning. Simply creating risky (in the colloquial meaning) things is not itself a reason to deserve money.

When you make markets you are literally paid the spread to assume the risk of holding the position.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#64

Post by gumby » Sun, Sep 10, 2023, 9:17 PM UTC

rkuykendall-com wrote: post #59

gumby wrote: post #51

> (why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point). Actually it's a big part of the point: they have a system that works with dead code in it. If you remove that dead code perhaps it unwittingly breaks something else. That kinds of chesterson's fence is a good practice.

Leaving dead code in is not good practice?? I would love more explanation here because that sounds like crazy talk to me.

You'll have to ask the author of the article.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#65

Post by envsubst » Sun, Sep 10, 2023, 9:17 PM UTC

UncleMeat wrote: post #34

Nevermark wrote: post #23

Earlier quoted context omitted.

Leeching implies someone has found a way to skim value from you without providing value. Someone taking on loads of risk to carry out your commands efficiently is providing value. You can argue whether they are doing so competently or not, or whether they are pricing optimally or not, but they are not just ”takers” or “leeches”.

How are they taking on loads of risk? Risk has a particular meaning in investment and "well, a bug can blow up my company" isn't part of that meaning. Simply creating risky (in the colloquial meaning) things is not itself a reason to deserve money.

> Risk has a particular meaning in investment

Risk in finance definitely takes on more meaning than the narrow definition in modern portfolio theory (stddev of price).

Re: Knightmare: A DevOps Cautionary Tale (2014)

#66

Post by envsubst » Sun, Sep 10, 2023, 9:19 PM UTC

hgomersall wrote: post #41

daft_pink wrote: post #39

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

I'm so glad I'm not wasting my life working in finance.

[flagged]

Re: Knightmare: A DevOps Cautionary Tale (2014)

#67

Post by uxp8u61q » Sun, Sep 10, 2023, 9:19 PM UTC

rvz wrote: post #53

But ChatGPT would have fixed the issue faster in 45 mins than a human would. /s A high risk situation like this would make the idea of using LLMs for this as not an option; before someone puts out a 'use-case' for a LLM to fix this issue. I'm sorry to preempt the thought of this in advance, but it would not.

Who are you replying to? Nobody but you talked about chatbots in this thread. Are you talking to yourself?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#68

Post by [deleted] » Sun, Sep 10, 2023, 9:20 PM UTC

rkuykendall-com wrote: post #59

gumby wrote: post #51

> (why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point). Actually it's a big part of the point: they have a system that works with dead code in it. If you remove that dead code perhaps it unwittingly breaks something else. That kinds of chesterson's fence is a good practice.

Leaving dead code in is not good practice?? I would love more explanation here because that sounds like crazy talk to me.

[deleted]

Re: Knightmare: A DevOps Cautionary Tale (2014)

#69

Post by fphhotchips » Sun, Sep 10, 2023, 9:20 PM UTC

gumby wrote: post #64

rkuykendall-com wrote: post #59

Earlier quoted context omitted.

Leaving dead code in is not good practice?? I would love more explanation here because that sounds like crazy talk to me.

You'll have to ask the author of the article.

Your original comment is somewhat unclear. Are you advocating for leaving old code in because the system works and it's more stable that way, or taking it out to force the necessary refactoring steps and understanding that will bring?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#70

Post by callalex » Sun, Sep 10, 2023, 9:20 PM UTC

daft_pink wrote: post #39

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

It’s fine to have that kind of responsibility, but it has to actually be your responsibility. Which means you have to be empowered to say “no, we aren’t shipping this until XYZ is fixed” even if XYZ will take another two years to build and the boss wants to ship tomorrow.