Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

31–40 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#31

Post by andersa »

ledauphin wrote: post #18
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

I wonder if this code was written in c++ or similar, the flags were actually a bitfield, and they repurposed it because they ran out of bits.

Need a space here? Oh, let's throw out this junk nobody used in 8 years and there we go...

Re: Knightmare: A DevOps Cautionary Tale (2014)

#32

Post by INTPenis »

40yearoldman wrote: post #24

lol. No. Deployments were not the issue. At any given time an automated deployment system could have had a mistake introduced that resulted in bad code being sent to the system. It does not matter if it was old or new code. Any code could have had this bug. What the issue was, and it’s one that I see often. Firstly no vision into the system. Not even a dash board showing the softwares running version. How often i see…

I mean it makes no sense, without even reading the article, just by working in IT I can tell you that if you're one deployment away from being bankrupt then you're either doing it wrong, or in the wrong business.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#33

Post by SoftTalker »

notnmeyer wrote: post #21
hyperhopper wrote: post #15

Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.

this is what stood out to me reading the story. i wonder if there was a reason why they opted for this, however half-baked. it reads less to me like a case for devops as it does a case for better practices at every stage of development. how arrogant or willfully ignorant do you have to be to operate like this considering what’s at stake?

They probably already had a bitfield of feature flags, maybe it was a 16-bit integer and full, and someone notices "hey this one is old, we can reuse it and not have to change the datatype"

Re: Knightmare: A DevOps Cautionary Tale (2014)

#34

Post by UncleMeat »

Nevermark wrote: post #23
loa_in_ wrote: post #19

Earlier quoted context omitted.

I don't see how anything here goes against it leeching off people

Leeching implies someone has found a way to skim value from you without providing value. Someone taking on loads of risk to carry out your commands efficiently is providing value. You can argue whether they are doing so competently or not, or whether they are pricing optimally or not, but they are not just ”takers” or “leeches”.

How are they taking on loads of risk? Risk has a particular meaning in investment and "well, a bug can blow up my company" isn't part of that meaning.

Simply creating risky (in the colloquial meaning) things is not itself a reason to deserve money.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#35

Post by Neil44 »

hyperhopper wrote: post #15

Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.

I can only think that it was some kind of fixed binary blob of 1/0 flags where all the positions had been used umpteen times over the years and nobody wanted to mess with the system to replace it with something better.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#36

Post by notnmeyer »

SoftTalker wrote: post #33
notnmeyer wrote: post #21

Earlier quoted context omitted.

this is what stood out to me reading the story. i wonder if there was a reason why they opted for this, however half-baked. it reads less to me like a case for devops as it does a case for better practices at every stage of development. how arrogant or willfully ignorant do you have to be to operate like this considering what’s at stake?

They probably already had a bitfield of feature flags, maybe it was a 16-bit integer and full, and someone notices "hey this one is old, we can reuse it and not have to change the datatype"

ah, yeah—hadn’t considered that!

Re: Knightmare: A DevOps Cautionary Tale (2014)

#37

Post by jraph »

Nevermark wrote: post #23
loa_in_ wrote: post #19

Earlier quoted context omitted.

I don't see how anything here goes against it leeching off people

Leeching implies someone has found a way to skim value from you without providing value. Someone taking on loads of risk to carry out your commands efficiently is providing value. You can argue whether they are doing so competently or not, or whether they are pricing optimally or not, but they are not just ”takers” or “leeches”.

What value do they provide?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#38

Post by SoftTalker »

thorum wrote: post #13

Honestly seems like the market itself should have safeguards against this kind of thing.

The safeguard is "you go bankrupt if you fuck up"

Imagine there was some way for a trading company to execute billions of dollars of trades and they say "ooops, sorry, that was all a mistake" can you not see how that would be abused?

Now, the story also says that within a minute of the market opening, the experienced traders knew something was wrong. Do they bear any culpability for jumping on those trades, making their money off of something they knew couldn't be intentional?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#40

Post by callalex »

jraph wrote: post #37
Nevermark wrote: post #23

Earlier quoted context omitted.

Leeching implies someone has found a way to skim value from you without providing value. Someone taking on loads of risk to carry out your commands efficiently is providing value. You can argue whether they are doing so competently or not, or whether they are pricing optimally or not, but they are not just ”takers” or “leeches”.

What value do they provide?

They claim to provide liquidity, even though they are just front-running trades that are already happening anyway.
Post reply on HN