Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

221–230 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#221

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

> the deployment agent errored while downloading the new binary/code onto the server

In that case the build would never be pushed to production. The worst it would accomplish, and this is if your systems fail, is that it will break your staging area.

Sure this is in the the ideal world where people actually know how to set up their deployment pipelines correctly, so you’re likely still right in many cases, but you shouldn’t be.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#222

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

Things are somewhat different now than 5, 10, 20 years ago.

There has been a wave of "individual accountability regimes" released by pretty much every regulator.

I have worked with the SFC the most, so that's what I will describe here, but all these regulations are pretty much copy/paste of each other anyway.

I was MIC under the SFC (HK) for various operational and financial resps for approx 8 years responsible for close to $3B exposure across equities, IRS & FX, and am now licensed with the FCA (UK) since approx 1 year.

Basically, on top of the usual regulatory framework defining a top level Operating Officer (MOO) and subordinate Responsible Officers (RO), the new individual accountability regime creates the notion of Managers In Charge (MIC).

The MICs fill the gap that, increasingly, a considerable amount of operational responsibility lie in the hands of non licensed individuals (i. e. tech people).

The SFC defines a number of responsabilites (e.g. DRP/BCP, kill switches, backups, fail overs, rollbacks, load testing, etc) and these responsibilites need to be allocated to one or more of the allocated MIC.

The SFC has a right to reject an appointment of MIC if the individual is not seen as fit and proper (that is assessed generally on an annual basis by a compliance officer, but can be re-assessed on the spot if you end up displaying unfit traits). The SFC also mandates a track record of experience and expertise on the assigned responsibilities, as well as a direct capability by the MIC to have control on his responsibilities. In clear terms, that means you need to have the actual power of saying "no", you need to have the power to hire someone if that is necessary for the safety of the operations, etc.

Once you get appointed as MIC, most of your responsibilities are based on _means_, not _end results_:

If Karen breaks production, that's not much of your problem (regulatorily speaking) as long as you can demonstrate that you had Karen attend 6h of training this year on how not to break production.

In terms of actual developer experience, the _means_ often take the form of trainings, code review, pre prod impact assessment, incident reporting procedures, etc.

So on one hand you have a very heavy personal and professional responsibility. But on the other end you are at fault only if you did not setup a proper framework for things to work.

In terms of the professional responsibility, there is not much to do if you are deemed guilty. You will most likely be temporarily or permanently barred from having a licensed position. Nobody will hire you anyway.

For the personal responsibility, it is usually limited to single digit millions, and most big asset managers have an insurance to protect you (otherwise noone would accept the role).

If you are interested in the actual additional responsibilities that were added after KC, then I suggest you have a look at MiFID II (the European régulation, well written and understandable), especially segment RTS 6 "Technical standards specifying the organisational requirements of investment firms engaged in algorithmic trading":

https://ec.europa.eu/finance/securities/docs/isd/mifid/rts/1...

Re: Knightmare: A DevOps Cautionary Tale (2014)

#224
post #172

Earlier quoted context omitted.

I'm so glad I'm not wasting my life working in finance.

Actually it's one of the few truly intellectually-pure endeavors. Everything else is the same pursuit with extra steps: Make a trading strategy to make money vs Make a cutting edge machine learning classifier to back out latent meaning in search queries to produce better search results to drive more traffic to google to sell ads to make money

You're not wrong, but the problem is those steps are also the steps that produce food, or improve health, or solve climate change or solve any of the innumerable problems we face as a society. As you identify, there are plenty of pursuits other than finance that are not particularly socially useful - it's not a very exclusive club.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#225

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

I always remember the Codeless Code on comments when I read about this: http://thecodelesscode.com/case/41?topic=comments

Re: Knightmare: A DevOps Cautionary Tale (2014)

#226
post #220
post #26

Earlier quoted context omitted.

> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill sw…

> Actually, we made way less of Knight's $400m than we could have because our risk systems kept shutting strategies down because what was happening was "too good to be true". Aren't a lot of trades undone anyway by the authorities after such severe market hiccups?

This is a good question. In my experience, I have only see exchange trades reversed when there was a major bug in exchange software. If the bug is on the client side, tough luck. And reversing trades done on an exchange is usually a decision for the exchange regulator. It is a major event that only happens every few years -- at most -- for highly developed exchanges.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#227
post #169

Earlier quoted context omitted.

They were asleep at the wheel, not unlike all the random brokerages that blew up when swiss central bank pulled the CHF peg in 2015. This is a culture problem - as soon as you load up your trading firm with a bunch of software industry hires, you end up with jiras and change management workflows instead of people on deck that have context for what they're doing. That's the only way to explain reverse scalping for 45…

The CHF de-peg wasn't really technology risk. Brokers lost money because they undervalued CHF/EUR risk, undervalued liquidity risk (stop orders were executing FAR worse than expected, or simply failing to execute at all), and didn't pay attention to the legal protections afforded to their customers (customer balances went negative but there was no way to recover that money from the customers). These brokers would hav…

I think it is worth saying that no one saw that de-peg coming. Absolutely no one. Sure there are some crazies who saw it coming, but that same camp is still taking for the HKD-USD de-peg. It was a shock to everyone on Wall Street. I am a bit surprised that the Swiss National Bank didn't tip off their own banks before doing it. Both UBS and Credit Suisse were seriously caught off guard when it happened.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#228

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…

> I've seen this same resistance to deleting code in programmers working in repos that are completely under VCS, and it's flabbergasting I think a lot of developers only know the basics of git. They can check in changes, they can look at history with git log, and maybe they know how to use git blame. They often don't know how to filter git history. They often don't know about the git pickaxe, or about exclude pattern…

> git pickaxe, or about exclude patterns, and don't even think to question if you can do something like "git log -G'int.*foo\(' -- ':(exclude)directory'" to search for 'foo' in the git log, excluding some directory

Learned something new today, thank you! Will find ways to use these in my daily workflow.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#229

Earlier quoted context omitted.

The CHF de-peg wasn't really technology risk. Brokers lost money because they undervalued CHF/EUR risk, undervalued liquidity risk (stop orders were executing FAR worse than expected, or simply failing to execute at all), and didn't pay attention to the legal protections afforded to their customers (customer balances went negative but there was no way to recover that money from the customers). These brokers would hav…

I think it is worth saying that no one saw that de-peg coming. Absolutely no one. Sure there are some crazies who saw it coming, but that same camp is still taking for the HKD-USD de-peg. It was a shock to everyone on Wall Street. I am a bit surprised that the Swiss National Bank didn't tip off their own banks before doing it. Both UBS and Credit Suisse were seriously caught off guard when it happened.

[deleted]

Re: Knightmare: A DevOps Cautionary Tale (2014)

#230
post #134

Earlier quoted context omitted.

CI/CD would have have solved this 100%: > ... one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Read this part again: > ... one of Knight’s technicians did not *copy the new code to…

Ansible in my experience will stop trying to run subsequent tasks on a server once one of them fails, but it will go ahead with other servers that match the inventory pattern. So it very well could have successfully updated 7 out of 8 hosts. Maybe there is a switch that will stop everything if any task on any host fails but it's not the default behavior. At least it would have logged an error that hopefully would hav…

> So it very well could have successfully updated 7 out of 8 hosts.

The problem was that the feature flag was manually enabled on the host with old code. Presumably with automated deployment the feature flag would never have been toggled if the deployment failed, either because the deployment didn't get that far or because the human spotted the failed deployment.

Post reply on HN