Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

71–80 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#71

Post by lionkor »

rkuykendall-com wrote: post #59
gumby wrote: post #51

> (why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point). Actually it's a big part of the point: they have a system that works with dead code in it. If you remove that dead code perhaps it unwittingly breaks something else. That kinds of chesterson's fence is a good practice.

Leaving dead code in is not good practice?? I would love more explanation here because that sounds like crazy talk to me.

It may not be obvious that it's dead code - in a lot of popular interpreted languages, it's impossible to tell if a given function can be called or not

Re: Knightmare: A DevOps Cautionary Tale (2014)

#72

Post by OtherShrezzing »

rkuykendall-com wrote: post #59
gumby wrote: post #51

> (why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point). Actually it's a big part of the point: they have a system that works with dead code in it. If you remove that dead code perhaps it unwittingly breaks something else. That kinds of chesterson's fence is a good practice.

Leaving dead code in is not good practice?? I would love more explanation here because that sounds like crazy talk to me.

Chesterton's Fence states that you shouldn't make a change until you understand something's current state. Removing code because it's dead is folly, if you don't understand 1) why it's there, and 2) why nobody else removed it yet.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#73

Post by [deleted] »

gumby wrote: post #51

> (why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point). Actually it's a big part of the point: they have a system that works with dead code in it. If you remove that dead code perhaps it unwittingly breaks something else. That kinds of chesterson's fence is a good practice.

[deleted]

Re: Knightmare: A DevOps Cautionary Tale (2014)

#74

Post by shric »

valdiorn wrote: post #27

Literally everyone in quant finance knows about knight capital. It even has its own phrase; "pulling a knight capital" (meaning; cutting corners on mission critical systems, even ones that can bankrupt the company in an instant, and experiencing the consequences)

Indeed, it's used in onboarding material at my employer.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#75

Post by lwhi »

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

Automated deployments require planning before the time they're executed.

If code is involved, someone likely reviews and approves it.

There are naturally far more safeguards in place than there would be for a manual deployment.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#76

Post by xyst »

Having worked in some Fortune 500 financial firms and low rent “fintech” upstarts, I am not surprised this happened. Decades of bandaid fixes, years of rotating out different consultants/contractors, and software rot. Plus years of emphasizing mid level management over software quality.

As other have mentioned, I don’t think “automation of deployment” would have prevented this company’s inevitable downfall. If it wasn’t this one incident in 2014, then it would have been another incident later on.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#77

Post by Nekhrimah »

ledauphin wrote: post #18
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

>whoever decided that reusing an old flag was a good idea.

My understanding is that in high frequency trading, minimizing the size of the transmission is paramount. Hence re-purposing an existing flag, rather than adding size to the packet makes some sense.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#78

Post by goldinfra »

envsubst wrote: post #66
hgomersall wrote: post #41

Earlier quoted context omitted.

I'm so glad I'm not wasting my life working in finance.

[flagged]

Most jobs do in fact contribute to the well being of humanity, however little. It's few jobs, like most in financial trading, that actively reduce the well being of humanity.

Never will you meet a more self-deluded and pathetic set of humans. Desperate money addicts that often become other kinds of addicts. Whole thing should be abolished.

Source: I worked in finance when I was young and dumb.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#79

Post by shric »

hgomersall wrote: post #41
daft_pink wrote: post #39

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

I'm so glad I'm not wasting my life working in finance.

I've worked in various small to medium IT companies, a FAANG and another fortune 500 tech company. 6 months ago I moved to a proprietary trading company/market maker and it's the most interesting and satisfying place I've worked so far.

I hope to continue to "waste my life" for many years to come.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#80

Post by bsagdiyev »

uxp8u61q wrote: post #67
rvz wrote: post #53

But ChatGPT would have fixed the issue faster in 45 mins than a human would. /s A high risk situation like this would make the idea of using LLMs for this as not an option; before someone puts out a 'use-case' for a LLM to fix this issue. I'm sorry to preempt the thought of this in advance, but it would not.

Who are you replying to? Nobody but you talked about chatbots in this thread. Are you talking to yourself?

No they're preempting someone coming along and claiming this. Haven't seen it in the replies yet but there's typically one (or a lot in some cases) person(s) claiming ChatGPT will bring Jesus back from the dead sort of thing.
Post reply on HN