[flagged]
How is high speed trading any more dubious than long term holding, or shorting, etc?
It’s all dubious.
Knightmare: A DevOps Cautionary Tale (2014)
131–140 of 294 posts
Re: Knightmare: A DevOps Cautionary Tale (2014)
#132Re: Knightmare: A DevOps Cautionary Tale (2014)
#133Earlier quoted context omitted.
It’s fine to have that kind of responsibility, but it has to actually be your responsibility. Which means you have to be empowered to say “no, we aren’t shipping this until XYZ is fixed” even if XYZ will take another two years to build and the boss wants to ship tomorrow.
Yep. Until the capacity to say unoverridably "No" materializes, there's a lot of code I refuse to have responsibility for delegated to me.
As a profit non-taker, which responsibility a worker can even have? Realistically it lies in range of their monthly paycheck and pending bonuses and in a moral obligation to operate a failing system until it lands somewhere. Everything above it is a systemic risk for a profit taker which if left unaddressed is absolutely on them. There’s no way you can take responsibility for $400M unless you have that money.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#134I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…
CI/CD would have have solved this 100%:
> ... one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added.
Read this part again:
> ... one of Knight’s technicians did not *copy the new code to one of the eight SMARS computer servers*.
Yes, of course a CI/CD pipeline can fail midway through and only partially deploy the code to a partial number of servers, but I doubt it. And even if that were the case, just off the top of my head I can guarantee an Ansible Playbook would have not only stopped the moment that particular transfer failed, the whole Playbook would have therefore failed, and none of the services would have been restarted (because that would be a final step that wouldn't be reached.)
This was due to human error and is the very reason CI/CD/automation is a thing.
> Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server
CI/CD would have have solved this 100%. A "Pull Request" made against a repository of Ansible code (or whatever you flavour is) would have *PREVENTED* the first technician from ever being able to merge the code into master/main (because you have master/main protected right... right?), completely preventing the entire process from ever rolling out without a review, which would have hopefully caught the misaligned configuration.
DevOps, which is mostly underpinned by CI/CD, would have solved this 100%. I'm very certain of this.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#135I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…
I see this as a problem of not investing enough in the deploy process. (Disclosure: I maintain an open source deploy tool for a living). Charity Majors gave a talk in Euruko that talked a lot about this. Deploy tooling shouldn’t be a bunch of bash scripts in a trench coat, it should be fully staffed, fully tested, and automated within an inch of its life. If you have a deploy process that has some king of immutable a…
Great reply.
Counterpoint though: that automation in and of itself is more failure area.
I can imagine a similar story where the deployment pipeline incorrectly rolled back due to some change in metric format and caused the infinite loss, for example.
The thing with these being a 1 in a million chance is that there's thousands of different hypothetical causes. The more parts the harder to predict an interaction and we've all been blindsided by something.
I would personally hate the stress of working on such high stakes releases.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#136Earlier quoted context omitted.
The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."
I wonder if this code was written in c++ or similar, the flags were actually a bitfield, and they repurposed it because they ran out of bits. Need a space here? Oh, let's throw out this junk nobody used in 8 years and there we go...
This sounds particularly plausible with it being high frequency trading. Those presumably have optimisations few other applications have
Re: Knightmare: A DevOps Cautionary Tale (2014)
#137Earlier quoted context omitted.
But how would you even start to define something as stochastic as trading activity as “not looking right”?
I’ve had to fill out forms for new algorithms / quant strategies with questions like: - how many orders per minute do you expect to create? - how many orders per minute do you expect to cancel/amend? - what’s your max per-ticker position? - what’s your max strategy-level GMV/NMV? Etc. Any one of those questions can be used to set up killswitches. [edited for formatting]
Sure, but there is always the possibility that then you shut down trading when things _arent_ broken.
There are always two error rates.
Defining behavior is great for retrospective analysis but would you really feel comfortable putting hard cuts into production based on the answers to those questions? I’m genuinely asking, because IME I wouldn’t be.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#138Earlier quoted context omitted.
But how would you even start to define something as stochastic as trading activity as “not looking right”?
spamming the market with orders for one
Define “spamming”, then? High frequency traders would probably look a lot like spammers.
There are always two error rates.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#139Earlier quoted context omitted.
The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."
At the very least have a two deploys - actually removing the old code that relies on it and then repurposing it. Giant foot gun to do it all in one especially without any automated deploys.
Good point. Actually I think I'll treat this as a best practice in general when there's a transition
Re: Knightmare: A DevOps Cautionary Tale (2014)
#140I don't exactly understand what this has to do with continuous delivery, but maybe I just don't know enough about this topic.
Wouldn't it have been best to set up a 'shadow infrastructure' and route every trade into it for several weeks/months to verify the correctness of the system?