Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

231–240 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#231

Post by blitzar »

dmurray wrote: post #26
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill sw…

I messed around with the idea of a physical big red button kill switch to shut down market making; the IT people thought I was joking - the trading desk just assumed that it was in the design from day 1.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#232

Post by dmurray »

eru wrote: post #220
dmurray wrote: post #26

Earlier quoted context omitted.

> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill sw…

> Actually, we made way less of Knight's $400m than we could have because our risk systems kept shutting strategies down because what was happening was "too good to be true". Aren't a lot of trades undone anyway by the authorities after such severe market hiccups?

Normally trades are undone or amended in price if they are executed far away (say 10%+) from what is determined to be reasonable market prices. And when amended, they get amended to a price that's still in the same direction, so the market taker still loses a little bit compared to the fair price.

KCG traded in such liquid instruments and in such a way that it didn't move the market that much. They lost a hundred dollars a trade on 4 million trades.

The article says some stocks were moved by more than 10%, but as I recall that was a small fraction of them.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#233

Post by dmurray »

throwaway2037 wrote: post #226
eru wrote: post #220

Earlier quoted context omitted.

> Actually, we made way less of Knight's $400m than we could have because our risk systems kept shutting strategies down because what was happening was "too good to be true". Aren't a lot of trades undone anyway by the authorities after such severe market hiccups?

This is a good question. In my experience, I have only see exchange trades reversed when there was a major bug in exchange software. If the bug is on the client side, tough luck. And reversing trades done on an exchange is usually a decision for the exchange regulator. It is a major event that only happens every few years -- at most -- for highly developed exchanges.

Reversing or amending a single "fat finger" trade happens all the time and the exchange generally has procedures for this that don't involve a regulator.

Even in the most controversial recent example - LME cancelling a day's worth of nickel trades [0]- I understand it was their call and not any external regulator. That said, while I'd count LME as a "highly developed exchange", it's the Wild West compared to the US NMS.

[0] https://www.bloomberg.com/news/articles/2022-03-14/inside-ni...

Re: Knightmare: A DevOps Cautionary Tale (2014)

#234

Post by jve »

TheDong wrote: post #197
lolinder wrote: post #177

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

> I've seen this same resistance to deleting code in programmers working in repos that are completely under VCS, and it's flabbergasting I think a lot of developers only know the basics of git. They can check in changes, they can look at history with git log, and maybe they know how to use git blame. They often don't know how to filter git history. They often don't know about the git pickaxe, or about exclude pattern…

> and don't even think to question if you can do something like "git log -G'int.*foo\(' -- ':(exclude)directory'"

I do question it, but I know the answer is hard (as you demonstrated), so I don't bother. And I'm now looking at git log docs - I fail to parse how the exclude works even looking at the docs - I don't find anything about `:(` contstruct which houses exclude keyword. But thanks for -G - that will be useful.

> Code in the git log is invisible to a lot of tooling

This is the issue. I expect when searching code to have a way to search older commits. But azure devops won't do it. There is no checkbox "Include all commits"

Re: Knightmare: A DevOps Cautionary Tale (2014)

#235

Post by BoorishBears »

dfinninger wrote: post #215
BoorishBears wrote: post #158

Earlier quoted context omitted.

I think this is an example of hindsight not always being 20/20 If you replace each step of the post mortem with a CI/CD based alternative, you miss out on the fact CI/CD trivializes designs where this wouldn't have happened. The "easy default" wouldn't be to run a play against 8 hosts manually in your terminal, it'd be run a playbook with them all baked in, and that would fail correctly by default: https://docs.ansib…

The easy default here would be to have a runbook that executed on a particular inventory group. The “linear” execution strategy is the default (which you linked to). By default, if there is an error on one host it will continue executing on all other hosts. You need to set a flag to stop executing on all hosts[1]. The parent process would not be notified of any failures until the end of the run, unless you supplied a…

The problem was a human forgot to run a step and no one noticed: The playbook would have failed and the server wouldn't have been online to make orders.

If you read the article, the other servers were fine and did not contribute to the issue.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#236

Post by pnt12 »

lolinder wrote: post #177

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

nitpick: git is enough to keep the history of old code, but it's not great at letting you find it.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#237

Post by eru »

dmurray wrote: post #233
throwaway2037 wrote: post #226

Earlier quoted context omitted.

This is a good question. In my experience, I have only see exchange trades reversed when there was a major bug in exchange software. If the bug is on the client side, tough luck. And reversing trades done on an exchange is usually a decision for the exchange regulator. It is a major event that only happens every few years -- at most -- for highly developed exchanges.

Reversing or amending a single "fat finger" trade happens all the time and the exchange generally has procedures for this that don't involve a regulator. Even in the most controversial recent example - LME cancelling a day's worth of nickel trades [0]- I understand it was their call and not any external regulator. That said, while I'd count LME as a "highly developed exchange", it's the Wild West compared to the US N…

> [...] the exchange generally has procedures for this that don't involve a regulator.

That's part of why I vaguely referred to 'the authorities' in my original comment. I wasn't quite sure who's doing the amending and reversing, and it wasn't too important.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#238

Post by jwestbury »

pxmpxm wrote: post #169
dmurray wrote: post #26

Earlier quoted context omitted.

> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill sw…

They were asleep at the wheel, not unlike all the random brokerages that blew up when swiss central bank pulled the CHF peg in 2015. This is a culture problem - as soon as you load up your trading firm with a bunch of software industry hires, you end up with jiras and change management workflows instead of people on deck that have context for what they're doing. That's the only way to explain reverse scalping for 45…

> as soon as you load up your trading firm with a bunch of software industry hires

As a software industry hire at a hedge fund right now... I'd love to see more cross-pollination, because there are so many good things happening on both sides, and so many terrible things happening through just a sheer lack of knowledge.

Change management workflows are great and should be used more in finance. But software companies should implement andon cord systems more often (Amazon does; nowhere else I've worked gives that power to anybody at the company).

Re: Knightmare: A DevOps Cautionary Tale (2014)

#239

Post by WJW »

hinkley wrote: post #113
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

The goal with automation is that the number of unidentified corner cases reduces over time. A manual runbook is a game of, "I did step 12, I think I did step 13, so the next step is 14." that plays out every single time you do server work. The thing with the human brain is that when you interrupt a task you've done a million times in the middle, most people can't reliably discern between this iteration and false memo…

1. Print the checklist/runbook out on paper with actual empty boxes next to the steps.

2. Laminate the printed checklist and put it in a big folder.

3. Every time you run the checklist, use a sharpie to mark the checkbox after you've done the step.

4. When you are done with the entire process, use whiteboard cleaner to wipe out the checks again and put the checklist back in the big folder with all the other checklists.

This is how every safety critical profession (aviation, shipping, medical, power generation, etc) has worked for decades and unless people are willingly being obtuse it is extremely hard to do it wrong. You just need people to turn off their ego and follow the process instead of trying to show off by doing it from memory. This last part might be more difficult in software settings.

Post reply on HN