Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

261–270 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#261

Post by alexpotato »

Couple fun facts/stories:

1. I signed my offer letter to work at Knight 5 days before this happened (and I still went to work there)

You can read more about that here: https://twitter.com/alexpotato/status/1501174282969305093

2. As I mentioned above, I went to work at Knight as a DevOps on a team that deal directly with the team mentioned in the blog post.

There are lots of stories around this but I will share this one:

Late 2012 is when Apple rolled out the "emergency weather notification" function. I was in the office and the notification went off on multiple people's phones. Knight was also experimenting with call notifications.

So when the alert goes off, someone yells "God damn it! Not again!!" (thinking there was another big outage)

3. People outside of finance have no idea of the different types of outage that can happen due to all sorts of factors.

I have a LOT of stories here: https://twitter.com/alexpotato/status/1215876962809339904

4. In finance in general, the amount of legacy code that behaves in weird ways or was written by someone 10 years ago who is no longer with the firm is ASTOUNDING.

Coupled with the billions of combinations of regulations, internal controls, multiple countries and jurisdictions etc makes accounting for every single edge case impossible. To use an infosec term the "attack surface" of possible user actions that could lead to bugs is enormous.

Typical case:

- User says they want to see reports for a couple days worth of trading for all securities

- User also says they want to see FULL history for one security

- User never says they might want to see FULL history for ALL securities at the same time

- This being HN, someone will say "you should have thought of that"

- Sure, but then they pull only some of the history for a Ukranian bond that has a 182 (not 180 like most) day bond. This is the only example of this type of bond. Ever. Did you think of that? What should the system have done?

- An oh, btw, this system was pushed out quickly due to regulatory pressure etc

Re: Knightmare: A DevOps Cautionary Tale (2014)

#262

Post by alexpotato »

lolinder wrote: post #177

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

See my sister thread where I worked at Knight just after the outage.

> For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag

There was another issue where they were using a database with only 256 columns. Sometimes, they needed a new column so they would just reuse an old columns that "wasn't being used at the time".

IIRC, this was generally acknowledged internally to be "a bad idea" but no one had prioritized cleaning up the old code and/or coming up with a better best practice.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#263

Post by Kinrany »

brazzy wrote: post #258
dmurray wrote: post #26

Earlier quoted context omitted.

> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill sw…

> or because no one was empowered to pull the kill switch because of the opportunity cost (perhaps pulling the switch at the wrong time costs $500k in opportunity). Isn't the problem that pulling the plug on a trading bot doesn't just have opportunity costs, but may also leave you with open positions that, depending on the kind of trades you're doing and the way the market is moving, could be arbitrarily expensive to…

Can be two buttons, one for buying and one for selling

Re: Knightmare: A DevOps Cautionary Tale (2014)

#264

Post by ycombobreaker »

throwaway2037 wrote: post #227
ycombobreaker wrote: post #193

Earlier quoted context omitted.

The CHF de-peg wasn't really technology risk. Brokers lost money because they undervalued CHF/EUR risk, undervalued liquidity risk (stop orders were executing FAR worse than expected, or simply failing to execute at all), and didn't pay attention to the legal protections afforded to their customers (customer balances went negative but there was no way to recover that money from the customers). These brokers would hav…

I think it is worth saying that no one saw that de-peg coming. Absolutely no one. Sure there are some crazies who saw it coming, but that same camp is still taking for the HKD-USD de-peg. It was a shock to everyone on Wall Street. I am a bit surprised that the Swiss National Bank didn't tip off their own banks before doing it. Both UBS and Credit Suisse were seriously caught off guard when it happened.

That's fair. I wasn't close enough to see how "surprising" it was. The point stands that it looks nothing like HFT active trading risk. Knight Capital created such amazing training material for the industry. What went wrong with their software, how many decisions or practices could have prevented or narrowed the risk. What went wrong at trade time, how they had a clear window to pull the plug but died due to indecision and inaction.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#265

Post by datadrivenangel »

lolinder wrote: post #177

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

Maybe they had engineering leadership mandates where creating new feature flags was more paperwork?

I've seen that happen, where 'new' features took a lot of justification, but 'bugfixes' were free.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#266

Post by btown »

schneems wrote: post #81
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

I see this as a problem of not investing enough in the deploy process. (Disclosure: I maintain an open source deploy tool for a living). Charity Majors gave a talk in Euruko that talked a lot about this. Deploy tooling shouldn’t be a bunch of bash scripts in a trench coat, it should be fully staffed, fully tested, and automated within an inch of its life. If you have a deploy process that has some king of immutable a…

> the ability to quickly rollback to a prior known good stage

This is vital, but it's often not sufficient just to roll back, say, to a known good Docker image. Database migrations may have occurred that dropped columns that the old code expects to exist; feature flags may need to be changed; multiple services may need to be rolled back individually; data may have accumulated under new assumptions that breaks old assumptions when old code is applied to that new data.

One of the really subtle wins of devops as a discipline is that by allowing/forcing application teams to take responsibility for deployment, they're more exposed to thinking how to solve these things in a maintainable way: for instance, breaking out complex "the meaning of our data is changing"-type changesets/data migrations into individually reversible stages, with the stages merged onto the production branch over the course of multiple days where analysis is done on error rates and live data.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#267

Post by kasey_junk »

brazzy wrote: post #258
dmurray wrote: post #26

Earlier quoted context omitted.

> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill sw…

> or because no one was empowered to pull the kill switch because of the opportunity cost (perhaps pulling the switch at the wrong time costs $500k in opportunity). Isn't the problem that pulling the plug on a trading bot doesn't just have opportunity costs, but may also leave you with open positions that, depending on the kind of trades you're doing and the way the market is moving, could be arbitrarily expensive to…

You usually have a procedure to close out open positions via an alternate system (including via phone).

Re: Knightmare: A DevOps Cautionary Tale (2014)

#268

Post by gumballindie »

bluelightning2k wrote: post #147
gumballindie wrote: post #22

> Had Knight implemented an automated deployment system – complete with configuration, deployment and test automation – the error that cause the Knightmare would have been avoided. Would it have been avoided though? Configuration, deployment and test automation mean nothing if they don't do what they are supposed to do. Regardless of how many tests you have, if you don't test for the right stuff it's all useless.

Yes. It would have. The specific part is configuration as code. So the config change (flag activation) and code change (flag calling) would have been synchronized. And there wouldn't have been one server of 8 with a different build for a meaningful time and also if it did fail to deploy on that one server it would have been obvious.

That's based on the assumption that someone would have thought about testing that particular flag for that particular scenario.

In my view this would only have been caught by a deployment to an identical copy of production, with running, simulated transactions, and high level funtional testing. Testing for each individual config value and scenario of where it may be used is playing whack a mole. Basically, I'd make a clone of prod, simulate everything that happends externaly (APIs, etc) and observe transaction KPIs and other high level business indicators. Testing for tech is insuring that the tech works, and sometimes that means testing that it's broken.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#269

Post by PH95VuimJjqBqy »

hinkley wrote: post #120
manicennui wrote: post #101

Earlier quoted context omitted.

9 times out of 10, I see developers making the mistakes that everyone seems to want to blame on non-technical people. There is a massive amount of software being written by people with a wide range of capabilities, and a large number of developers never master the basics. It doesn't help that some of the worst tools "win" and offer little protection against many basic mistakes.

For a group who so thoroughly despises bosses that operate on 'blame allocation', we spend a lot of time shopping around for permission to engage in reckless behavior. Most people would call that being a hypocrite. Whereas I would call it... no, hypocrite works just fine.

At the company I work, we have a team that took 3 weeks and multiple tries to get an API response (JSON) capitalized properly (camelCase to PascalCase).

When I tried to talk to the tech lead about it, his response is that SAFe would have prevented the issue (it was discovered by another team who consumes their API).

Throughout the entire thing this tech lead maintained his team didn't do anything and that the problem was the process.

yeah, no. I have 25+ years of experience as a developer, it doesn't take 3+ weeks to fix the casing of a JSON property name. I eventually had to be the bad guy and tell them their work was unacceptable because they themselves couldn't recognize it. Only when I did it, I ran it up the chain because if the tech lead doesn't see the problem then I need someone who can help them see the problem.

For some people there's a "responsibility shield" that's so strong you can never get through to them.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#270

Post by lukeasrodgers »

alexpotato wrote: post #261

Couple fun facts/stories: 1. I signed my offer letter to work at Knight 5 days before this happened (and I still went to work there) You can read more about that here: https://twitter.com/alexpotato/status/1501174282969305093 2. As I mentioned above, I went to work at Knight as a DevOps on a team that deal directly with the team mentioned in the blog post. There are lots of stories around this but I will share this o…

I would be interested to read these stories, but the twitter links only show a single tweet ending in the phrase "A thread." Perhaps this is a new feature of X whereby only logged-in users can see a tweet and its replies.
Post reply on HN