21–30 of 243 posts
"The best part is the fine: $12m, despite the resulting audit also revealing that the system was systematically sending naked shorts." Cool - all you have to do to get away with financial crimes is create a system with no protections against breaking the law.
The week after this we had a trader in our office who had a meeting at Knight on the morning it happened. He said he saw the whole dev team just power off and go home at 11am, followed quickly by the rest of the employees. At that point, there was nothing they could do. The craziest thing is that it went on for so long. No one caught it until their own traders so it come across Bloomberg and CNBC. They actually thoug…
Seems like as a rule, they're likely to cause instability, and I have a hard time seeing any benefits in them.
Earlier quoted context omitted.
Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years , is obviously bad. Deploying in such a way that all your servers are not running the same codebase is obviously bad. Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad. Not having anyone monitor things closely enough, including the hundreds of…
I'm not sure I agree with "Deploying in such a way that all your servers are not running the same codebase is obviously bad." I have a lot of experience in large scale systems (although this incident with 8 machines does not qualify) and I would say there is _always_ a period of transition where versions X and Y are online in production simultaneously. How can it be otherwise? You'd need scheduled downtime to do it a…
Trading floor is only open a few hours every day, the functionality being rolled out required the markets to be open. Furthermore, since the changes were all for new functionality they rolled it out in stages days ahead of time (good move).
Earlier quoted context omitted.
Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years , is obviously bad. Deploying in such a way that all your servers are not running the same codebase is obviously bad. Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad. Not having anyone monitor things closely enough, including the hundreds of…
I wonder how you can have code sit on a server unused and then 8 years later have it be called? What language was this written in?
What sometimes happens when you want to decommission features is you just turn the flags off rather than remove the code. There's an obvious allure to this as you have already tested the on/off functionality of the switch when you did the original roll-out so you can avoid having to test whether you have removed the code correctly. It sounds like in this case they removed the code and repurposed the switch that disabled said code (it may really be a shared memory system and they are running out of flags), but they fucked it up. The old code was still there on some servers and the switch was turned back in intent to enable the new feature it was re-purposed for, re-enabling this old code.
Powder Keg is a distinctly un-reassuring name for finance related functionality.
It is peg, not keg. Peg refers to an order where the limit price is automatically adjusted to some benchmark. For instance, you always want to be 1 penny away from the best bid. I don't know specifically what "power peg" is, though.
Powder Keg is a distinctly un-reassuring name for finance related functionality.
It is peg, not keg. Peg refers to an order where the limit price is automatically adjusted to some benchmark. For instance, you always want to be 1 penny away from the best bid. I don't know specifically what "power peg" is, though.
Just one of the risks of automation, and a good reminder why human monitoring is necessary. Having said that, we deployed a system that was mostly automated, with the human operator to oversee investments and if any out-of-the-ordinary transactions (based on experience) were taking place, to shut it down. She happily sat there approving the recommendations even though the recommendations were absolutely outside of an…
You should read the linked PDF - they had systems that were 100% dependent on human monitoring, that no one was checking, or where no one recognized anything unusual. If anything, their failures were due to massive lack of automation in deployment, testing, and monitoring.
Powder Keg is a distinctly un-reassuring name for finance related functionality.
It is peg, not keg. Peg refers to an order where the limit price is automatically adjusted to some benchmark. For instance, you always want to be 1 penny away from the best bid. I don't know specifically what "power peg" is, though.