Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

211–220 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#211

Post by amluto »

distortionfield wrote: post #181
amluto wrote: post #165

Earlier quoted context omitted.

That last nine in a trading system uptime has exponentially low value unless you have customers who care quite a lot. Seriously, suppose you have a truly awesome system making $100B per year of revenue. If you unnecessarily shut down 0.1% of the time, that’s only $100M per year lost, and an 0.1% unnecessary shutdown rate seems pretty high.

> That last nine in a trading system uptime has exponentially low value IME that last 9 is where all the action happens > unless you have customers who care quite a lot All customers care about their trades. I’ve worked with these systems. You can’t treat smaller traders as less-than. > only $100M How far removed from the problem do you have to be to think one hundred million dollars is not going to effect anyone?

Not all automated trading systems have customers.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#212

Post by mirekrusin »

lolinder wrote: post #177

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

The morale should be "don't reuse feature flags kids".

Re: Knightmare: A DevOps Cautionary Tale (2014)

#213

Post by leaflets2 »

amluto wrote: post #43
hyperhopper wrote: post #15

Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.

I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.

Yes definitely, one has to assume that from time to time, bugs will reach the prod servers, no amount of tests and code review can completely prevent that.

Hopefully the kill switch system is reasonably easy to code review and test :-)

Re: Knightmare: A DevOps Cautionary Tale (2014)

#214

Post by dfinninger »

SoftTalker wrote: post #155
movedx wrote: post #134

Earlier quoted context omitted.

CI/CD would have have solved this 100%: > ... one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Read this part again: > ... one of Knight’s technicians did not *copy the new code to…

Ansible in my experience will stop trying to run subsequent tasks on a server once one of them fails, but it will go ahead with other servers that match the inventory pattern. So it very well could have successfully updated 7 out of 8 hosts. Maybe there is a switch that will stop everything if any task on any host fails but it's not the default behavior. At least it would have logged an error that hopefully would hav…

Yeah, it’s a runbook config: https://docs.ansible.com/ansible/2.8/user_guide/playbooks_er...

Re: Knightmare: A DevOps Cautionary Tale (2014)

#215

Post by dfinninger »

BoorishBears wrote: post #158
SoftTalker wrote: post #155

Earlier quoted context omitted.

Ansible in my experience will stop trying to run subsequent tasks on a server once one of them fails, but it will go ahead with other servers that match the inventory pattern. So it very well could have successfully updated 7 out of 8 hosts. Maybe there is a switch that will stop everything if any task on any host fails but it's not the default behavior. At least it would have logged an error that hopefully would hav…

I think this is an example of hindsight not always being 20/20 If you replace each step of the post mortem with a CI/CD based alternative, you miss out on the fact CI/CD trivializes designs where this wouldn't have happened. The "easy default" wouldn't be to run a play against 8 hosts manually in your terminal, it'd be run a playbook with them all baked in, and that would fail correctly by default: https://docs.ansib…

The easy default here would be to have a runbook that executed on a particular inventory group.

The “linear” execution strategy is the default (which you linked to). By default, if there is an error on one host it will continue executing on all other hosts. You need to set a flag to stop executing on all hosts[1].

The parent process would not be notified of any failures until the end of the run, unless you supplied a custom callback plugin[2].

[1] https://docs.ansible.com/ansible/2.8/user_guide/playbooks_er...

[2] https://docs.ansible.com/ansible/latest/plugins/callback.htm...

Re: Knightmare: A DevOps Cautionary Tale (2014)

#216

Post by leaflets2 »

distortionfield wrote: post #137
Jorge1o1 wrote: post #130

Earlier quoted context omitted.

I’ve had to fill out forms for new algorithms / quant strategies with questions like: - how many orders per minute do you expect to create? - how many orders per minute do you expect to cancel/amend? - what’s your max per-ticker position? - what’s your max strategy-level GMV/NMV? Etc. Any one of those questions can be used to set up killswitches. [edited for formatting]

Sure, but there is always the possibility that then you shut down trading when things _arent_ broken. There are always two error rates. Defining behavior is great for retrospective analysis but would you really feel comfortable putting hard cuts into production based on the answers to those questions? I’m genuinely asking, because IME I wouldn’t be.

A way to add limits when being clueless:

Estimate what a real human can do in a day, and use that as the limits. Verify that the system behaves ok for some time, then scale up the desired trading volume and limits, observe, scale, repeat.

But you don't do it by making a (bad) guess up front and then just leaving it at that.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#217

Post by stevage »

>They had 48-hours to raise the capital necessary to cover their losses (which they managed to do with a $400 million investment from around a half-dozen investors).

I'm very curious about this bit. How exactly do you raise $400m of "investment" to cover such a massive footgun, in 48 hours, when you haven't even had time to understand what happened or whether it would happen again?

Why are people stumping up hundreds of millions of cash here?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#218

Post by IshKebab »

SamuelAdams wrote: post #183
lolinder wrote: post #177

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

Version control isn’t bulletproof either. All your code history is one git rebase away from being abolished forever. I hope most orgs have processes around their main branches so this does not occur, but I’ve also been in smaller orgs and accidentally screwed up prod database tables, so the accidental git rebase isn’t impossible to consider…

You're very unlikely to lose your version history like that.

Everywhere I've worked has branch protection turned on, and backups. Even if those both fail somehow it's very likely the complete history is on lots of engineers' laptops.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#219

Post by leaflets2 »

civilized wrote: post #150

I see a lot of criticism of the deployment, but why did the developers "repurpose an old flag" that activates 8 years dead code that you haven't deleted and that has completely unknown current functionality? That seems like the strangest decision made in this debacle.

To save time, I guess. They deleted the inactive code, so, why not, they thought. But then they forgot to deploy that change (to one server).

Bugs and configuration errors will happen from time to time, and might look silly in retrospect. But the real problem was, I think, that there was no kill switch (managers and tech leads should have decided to add long ago)

Re: Knightmare: A DevOps Cautionary Tale (2014)

#220

Post by eru »

dmurray wrote: post #26
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill sw…

> Actually, we made way less of Knight's $400m than we could have because our risk systems kept shutting strategies down because what was happening was "too good to be true".

Aren't a lot of trades undone anyway by the authorities after such severe market hiccups?

Post reply on HN