Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to lose $172,222 a second for 45 minutes

pythonsweetness.tumblr.com

Post reply on HN
Print view View original

11–20 of 243 posts

Re: How to lose $172,222 a second for 45 minutes

#11

Post by malbs »

Just one of the risks of automation, and a good reminder why human monitoring is necessary.

Having said that, we deployed a system that was mostly automated, with the human operator to oversee investments and if any out-of-the-ordinary transactions (based on experience) were taking place, to shut it down. She happily sat there approving the recommendations even though the recommendations were absolutely outside of anything we'd ever generated in the past, and bled accounts dry in one evening, so sometimes even with a human observing you're still boned.

Re: How to lose $172,222 a second for 45 minutes

#13

Post by coldcode »

MBCook wrote: post #10
shtylman wrote: post #3

Hindsight is 20/20

Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years , is obviously bad. Deploying in such a way that all your servers are not running the same codebase is obviously bad. Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad. Not having anyone monitor things closely enough, including the hundreds of…

I wonder how you can have code sit on a server unused and then 8 years later have it be called? What language was this written in?

Re: How to lose $172,222 a second for 45 minutes

#14

Post by frankc »

meepmorp wrote: post #6

Powder Keg is a distinctly un-reassuring name for finance related functionality.

It is peg, not keg. Peg refers to an order where the limit price is automatically adjusted to some benchmark. For instance, you always want to be 1 penny away from the best bid. I don't know specifically what "power peg" is, though.

Re: How to lose $172,222 a second for 45 minutes

#15

Post by thrownaway2424 »

MBCook wrote: post #10
shtylman wrote: post #3

Hindsight is 20/20

Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years , is obviously bad. Deploying in such a way that all your servers are not running the same codebase is obviously bad. Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad. Not having anyone monitor things closely enough, including the hundreds of…

I'm not sure I agree with "Deploying in such a way that all your servers are not running the same codebase is obviously bad." I have a lot of experience in large scale systems (although this incident with 8 machines does not qualify) and I would say there is _always_ a period of transition where versions X and Y are online in production simultaneously. How can it be otherwise? You'd need scheduled downtime to do it any other way.

I think the main problem here is nobody at this company pushed back on this stupid development plan of reusing a flag for a different purpose. There's no excuse for that (or maybe there is, they had run out of fields in some fixed-width message format or something dumb like that). Also apparently the use of the flag was not tied hermetically to the binary in production; when they rolled back the binary the flag was still there but it meant something different to the old software.

The correct way to roll this type of change out is for the new input (the "flag" in this case) to be totally inert for the old version of the software, and for the new version to have a config file or command line argument that disables it. So _first_ you start sending this new feature in the input, which is meaningless and ignored by the existing software, and then you roll out the new software to maybe 1% of your fleet, and see if it works. Then roll it out to maybe 10% and leave it that way for a week. Insist that your developers have created a a way to cross-check the correctness of the feature in the 10% test fleet (structured logging etc). If it looks good roll it to 100%. You now have three ways to disable it: turn it off in the input stream, turn it off in the new software with the config or argument, or roll back the software.

Doesn't look like these guys really knew what they were doing.

Re: How to lose $172,222 a second for 45 minutes

#16

Post by yareally »

coldcode wrote: post #13
MBCook wrote: post #10

Earlier quoted context omitted.

Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years , is obviously bad. Deploying in such a way that all your servers are not running the same codebase is obviously bad. Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad. Not having anyone monitor things closely enough, including the hundreds of…

I wonder how you can have code sit on a server unused and then 8 years later have it be called? What language was this written in?

Why would the language matter? Dormant code is the same in any language.

Re: How to lose $172,222 a second for 45 minutes

#17

Post by fiatmoney »

malbs wrote: post #11

Just one of the risks of automation, and a good reminder why human monitoring is necessary. Having said that, we deployed a system that was mostly automated, with the human operator to oversee investments and if any out-of-the-ordinary transactions (based on experience) were taking place, to shut it down. She happily sat there approving the recommendations even though the recommendations were absolutely outside of an…

You should read the linked PDF - they had systems that were 100% dependent on human monitoring, that no one was checking, or where no one recognized anything unusual. If anything, their failures were due to massive lack of automation in deployment, testing, and monitoring.

Re: How to lose $172,222 a second for 45 minutes

#18

Post by vinceguidry »

Narkov wrote: post #12

Out of interest, what would have been the outcome for Knight if their positions had caused them to be winners? $12m fine, keep the spoils and "carry on" ?

It's not really possible to fail upwards this way. It would be like forgetting how to play chess in the middle of a game and then winning. Anomalies are universally negative in high-stakes environments, or if they're positive, only engender modest improvements.

Re: How to lose $172,222 a second for 45 minutes

#19

Post by yogo »

coldcode wrote: post #13
MBCook wrote: post #10

Earlier quoted context omitted.

Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years , is obviously bad. Deploying in such a way that all your servers are not running the same codebase is obviously bad. Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad. Not having anyone monitor things closely enough, including the hundreds of…

I wonder how you can have code sit on a server unused and then 8 years later have it be called? What language was this written in?

It was a flag that was re-purposed. If the code is never deleted and the flag's use is discontinued then the code never runs. However, it seems that the flag was reused years later causing the old code to be inadvertently reactivated. It has nothing to do with the language it was written in.

Re: How to lose $172,222 a second for 45 minutes

#20

Post by [deleted] »

coldcode wrote: post #13
MBCook wrote: post #10

Earlier quoted context omitted.

Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years , is obviously bad. Deploying in such a way that all your servers are not running the same codebase is obviously bad. Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad. Not having anyone monitor things closely enough, including the hundreds of…

I wonder how you can have code sit on a server unused and then 8 years later have it be called? What language was this written in?

[deleted]
Post reply on HN