Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

101–110 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#101

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

9 times out of 10, I see developers making the mistakes that everyone seems to want to blame on non-technical people. There is a massive amount of software being written by people with a wide range of capabilities, and a large number of developers never master the basics. It doesn't help that some of the worst tools "win" and offer little protection against many basic mistakes.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#103

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

At the very least have a two deploys - actually removing the old code that relies on it and then repurposing it. Giant foot gun to do it all in one especially without any automated deploys.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#105
post #93

Earlier quoted context omitted.

The thing I was surprised about is that they survived! After all that they still got a $400 million cash bailout!

It wasn't a bailout, it was investment money. Bailout could imply government throwing a lifeguard

> It wasn't a bailout, it was investment money.

Almost all bailouts are investments, whether it's a government or private bailout.

The investments are questionable sometimes, but they're still investments.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#107

Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.

There's definitely more to this story. Why was there a fixed number of "flags" so that they needed to be reused? I wish there was a true technical explanation.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#109
post #93

Earlier quoted context omitted.

It wasn't a bailout, it was investment money. Bailout could imply government throwing a lifeguard

> It wasn't a bailout, it was investment money. Almost all bailouts are investments, whether it's a government or private bailout. The investments are questionable sometimes, but they're still investments.

The nuance is a) what happens to existing equity stakeholders and b) does the bailout have to be repaid.

If the answer is nothing and no, then it’s a bailout philosophically. If the existing investors get diluted then they’re in part paying for the new capital injection.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#110
post #43

Earlier quoted context omitted.

I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.

But how would you even start to define something as stochastic as trading activity as “not looking right”?

spamming the market with orders for one
Post reply on HN