Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

91–100 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#91
post #43

Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.

I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.

But how would you even start to define something as stochastic as trading activity as “not looking right”?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#92

Earlier quoted context omitted.

[flagged]

Most jobs do in fact contribute to the well being of humanity, however little. It's few jobs, like most in financial trading, that actively reduce the well being of humanity. Never will you meet a more self-deluded and pathetic set of humans. Desperate money addicts that often become other kinds of addicts. Whole thing should be abolished. Source: I worked in finance when I was young and dumb.

> Most jobs do in fact contribute to the well being of humanity, however little.

No, they don't. A lot of jobs hold os back, actually. Salespeople selling things people don't wanna buy, finance and tech bros vampirizing third world countries without the safeguards that western countries have on their capital markets, etc.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#93
post #76

Having worked in some Fortune 500 financial firms and low rent “fintech” upstarts, I am not surprised this happened. Decades of bandaid fixes, years of rotating out different consultants/contractors, and software rot. Plus years of emphasizing mid level management over software quality. As other have mentioned, I don’t think “automation of deployment” would have prevented this company’s inevitable downfall. If it was…

The thing I was surprised about is that they survived! After all that they still got a $400 million cash bailout!

It wasn't a bailout, it was investment money.

Bailout could imply government throwing a lifeguard

Re: Knightmare: A DevOps Cautionary Tale (2014)

#94

Earlier quoted context omitted.

Leaving dead code in is not good practice?? I would love more explanation here because that sounds like crazy talk to me.

Chesterton's Fence states that you shouldn't make a change until you understand something's current state. Removing code because it's dead is folly, if you don't understand 1) why it's there, and 2) why nobody else removed it yet.

As this is a postmortem, it was proven dead code. There is nothing in the text that mentions that they didn't know what the code did (which then wouldn't be dead code).

Re: Knightmare: A DevOps Cautionary Tale (2014)

#95
post #76

Having worked in some Fortune 500 financial firms and low rent “fintech” upstarts, I am not surprised this happened. Decades of bandaid fixes, years of rotating out different consultants/contractors, and software rot. Plus years of emphasizing mid level management over software quality. As other have mentioned, I don’t think “automation of deployment” would have prevented this company’s inevitable downfall. If it was…

The thing I was surprised about is that they survived! After all that they still got a $400 million cash bailout!

[deleted]

Re: Knightmare: A DevOps Cautionary Tale (2014)

#96
post #75

Earlier quoted context omitted.

Automated deployments require planning before the time they're executed. If code is involved, someone likely reviews and approves it. There are naturally far more safeguards in place than there would be for a manual deployment.

In an ideal world, sure. In the current one, we have Facebook's "Move fast and break things" being applied to many things where it has no business being. Banking and communications infrastructure comes to my mind, but there are definitely others. :)

I think the benefits from automated deployments are things that are just par for the course to be honest.

Sure, you can mess these things up .. but doing so would involve willful negligence rather than someone's absent mindedness.

Basically, I think the takeaway from the article is probably worth taking.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#97
post #64

Earlier quoted context omitted.

You'll have to ask the author of the article.

Your original comment is somewhat unclear. Are you advocating for leaving old code in because the system works and it's more stable that way, or taking it out to force the necessary refactoring steps and understanding that will bring?

I'm sorry I wasn't clear: I re-read my comment and couldn't think of a decent edit.

It was the author whom I was quoting as saying "why would someone have old code lying around." It seems obvious why that's a good idea and it seems commenters in this thread (including you) agree with me and not the author.

Sorry again if I was unclear.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#98
post #76

Having worked in some Fortune 500 financial firms and low rent “fintech” upstarts, I am not surprised this happened. Decades of bandaid fixes, years of rotating out different consultants/contractors, and software rot. Plus years of emphasizing mid level management over software quality. As other have mentioned, I don’t think “automation of deployment” would have prevented this company’s inevitable downfall. If it was…

The thing I was surprised about is that they survived! After all that they still got a $400 million cash bailout!

It wasn’t a bail out, it was an opportunity for investors to get great terms on equity at a crucial juncture for the company.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#99
post #76

Having worked in some Fortune 500 financial firms and low rent “fintech” upstarts, I am not surprised this happened. Decades of bandaid fixes, years of rotating out different consultants/contractors, and software rot. Plus years of emphasizing mid level management over software quality. As other have mentioned, I don’t think “automation of deployment” would have prevented this company’s inevitable downfall. If it was…

The thing I was surprised about is that they survived! After all that they still got a $400 million cash bailout!

They were probably worth much more than $400M before the failure so it was a good investment opportunity. They would have been a money printing machine aside from this one major fuckup.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#100
post #98

Earlier quoted context omitted.

The thing I was surprised about is that they survived! After all that they still got a $400 million cash bailout!

It wasn’t a bail out, it was an opportunity for investors to get great terms on equity at a crucial juncture for the company.

A government bail out isn't the exclusive use of the phrase "bail out", it was both a bail out and an opportunity for investors to get great terms on equity.
Post reply on HN