Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Reinforcement Learning in the Algorithmic Trading Problem

arxiv.org

Post reply on HN
Print view View original

41–50 of 139 posts

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#41

Post by marktangotango »

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

John von Neumann had a similar observation:

https://www.johndcook.com/blog/2011/06/21/how-to-fit-an-elep...

Edit down voters care to elaborate?

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#42

Post by traK6Dcm »

huac wrote: post #40
traderthrow454 wrote: post #20

Earlier quoted context omitted.

If you want to read useful academic papers about trading there is one author in particular who is actually not bad - Zura Kakushadze. Most of his stuff is applicable to mid-frequency trading, not HFT. He worked at WorldQuant (reputable trading firm) and the founder of WQ, Igor Tulchinsky, is a coauthor on one of his papers. Example of a pretty interesting and accessible one - is "101 Formulaic Alphas" [0]. [0] - http…

picking a random one out of the pile: > Alpha#90: ((rank((close - ts_max(close, 4.66719)))^Ts_Rank(correlation(IndNeutralize(adv40, IndClass.subindustry), low, 5.38375), 3.21856)) * -1) I wonder how these magic numbers get picked (4.66719, 5.38375, etc) -- I guess there is some optimization solver which attempts to find the most profitable variables for a given alpha formulation, but isn't this approach also very vul…

Yup, it's probably just the output of an optimizer and then tested on held-out future data. Not overfitting is the key here and what's really hard. You need to be careful about the number of parameters and the amount of validation data you have.

These alphas will likely be only profitable for a short time period as long as the market data distribution (i.e. strategies of other market participants) doesn't change. So you would need to continually optimize and update them.

The way I think about it is that you are essentially finding the right parameters to "exploit" the combination of algorithms of all other participants, where algorithm could also be a human looking at charts and following certain rules, with a lot of random noise from retail traders thrown in.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#44

Post by young_rutabaga »

traderthrow454 wrote: post #20
traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

If you want to read useful academic papers about trading there is one author in particular who is actually not bad - Zura Kakushadze. Most of his stuff is applicable to mid-frequency trading, not HFT. He worked at WorldQuant (reputable trading firm) and the founder of WQ, Igor Tulchinsky, is a coauthor on one of his papers. Example of a pretty interesting and accessible one - is "101 Formulaic Alphas" [0]. [0] - http…

This paper is a hilarious dump of WQ's randomly generated formulas that (hopefully) happen to pass in-sample test.

  Alpha#33: rank((-1 * ((1 - (open / close))^1)))
This formula trivially reduces to

  rank(open/close - 1)
which is an example of a mean-reversion strategy. But: 1) nobody bothered to simplify this formula, 2) as any mean reversion, it is extremely difficult to trade.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#45

Post by laxatives »

huac wrote: post #40
traderthrow454 wrote: post #20

Earlier quoted context omitted.

If you want to read useful academic papers about trading there is one author in particular who is actually not bad - Zura Kakushadze. Most of his stuff is applicable to mid-frequency trading, not HFT. He worked at WorldQuant (reputable trading firm) and the founder of WQ, Igor Tulchinsky, is a coauthor on one of his papers. Example of a pretty interesting and accessible one - is "101 Formulaic Alphas" [0]. [0] - http…

picking a random one out of the pile: > Alpha#90: ((rank((close - ts_max(close, 4.66719)))^Ts_Rank(correlation(IndNeutralize(adv40, IndClass.subindustry), low, 5.38375), 3.21856)) * -1) I wonder how these magic numbers get picked (4.66719, 5.38375, etc) -- I guess there is some optimization solver which attempts to find the most profitable variables for a given alpha formulation, but isn't this approach also very vul…

Seems kind of rudimentary. Namely

> (sign(delta(volume, 1)) * (-1 * delta(close, 1)))

That's crazy. Would be interesting to see WTF a "mega-alpha" actually does using these strategies.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#46

Post by smdz »

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

> In addition to that, people who are actually "good" at trading don't publish papers, they silently make money.

Well, that is mostly true. But never discount anything. There are people like me who used to love the data analysis and prediction part in these markets. I got hooked to the markets because of it. I was not interested in making money and naively thought my average pay was good enough. When I first built (or my machine built) a working strategy (in early 2008), live traded/tested it for a couple of months and told few colleagues about the details about the strategy - they did not take me seriously. This was even before I understood NNs or any of scikit-learn tooling. I knew I wanted to get into financial markets - went to a broker to sell the automated strategy and seeking a full time job as an algo-trader - they thought I was trying to scam them even after seeing the contract notes. Plus algotrading had not picked up back then. I found later about such scams. It took me 3 more years and a financial crisis to understand the value of making "much more than enough" money. And retrospectively I know those were just stupid attempts trying to convince others and attempting to give it away.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#47

Post by hhmc »

henning wrote: post #3

It looks like their code is assuming a fixed commission of 20 rubles, which is apparently equivalent to US$0.27. Does anyone know if that's realistic? https://github.com/evgps/a3c_trading/blob/master/configs.py#... Depending on how liquid the market they studied is, code that assumes there is never any slippage may not be very realistic. There's no comparison to a simple buy-and-hold strategy, which may be less inter…

IIRC moex is perticularly expensive to trade on, and costs can be non linear -- but something like 1bps of commms is a reasonable approximation as an upper bound. In the paper they claim a cost of 2.5RUB per transaction, not the 20 in the config file.

Edit*: It looks like the comms are indeed around 10RUB per side (which is approx 1bps). https://www.moex.com/en/contract.aspx

If the model is trading single lots then this is a reasonable cost assumption, otherwise it isn't. The paper using 2.5RUB as costs is unreasonable.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#48

Post by traK6Dcm »

smdz wrote: post #46
traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

> In addition to that, people who are actually "good" at trading don't publish papers, they silently make money. Well, that is mostly true. But never discount anything. There are people like me who used to love the data analysis and prediction part in these markets. I got hooked to the markets because of it. I was not interested in making money and naively thought my average pay was good enough. When I first built (o…

You make a good point. I've also gotten into trading because I enjoy the algorithmic and mathematical aspects, and I would love to share more of what has been working for me and write extensively about it. And there are probably more people like that out there. However, trading has such a bad reputation and uncertain future that I am not sure that's a good career move. I'm torn.

You're right that there are probably some gems and people writing up good posts and articles. However, 99% of what comes to my inbox, which is certain newsletters and arXiv subscriptions, is clearly BS. I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post. In trading the opposite seems to be true. Seeing a good trading paper on arXiv is incredibly rare. I would even go as far as saying that reddit is a significantly better source of information than arXiv for this field.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#49

Post by smdz »

huac wrote: post #40
traderthrow454 wrote: post #20

Earlier quoted context omitted.

If you want to read useful academic papers about trading there is one author in particular who is actually not bad - Zura Kakushadze. Most of his stuff is applicable to mid-frequency trading, not HFT. He worked at WorldQuant (reputable trading firm) and the founder of WQ, Igor Tulchinsky, is a coauthor on one of his papers. Example of a pretty interesting and accessible one - is "101 Formulaic Alphas" [0]. [0] - http…

picking a random one out of the pile: > Alpha#90: ((rank((close - ts_max(close, 4.66719)))^Ts_Rank(correlation(IndNeutralize(adv40, IndClass.subindustry), low, 5.38375), 3.21856)) * -1) I wonder how these magic numbers get picked (4.66719, 5.38375, etc) -- I guess there is some optimization solver which attempts to find the most profitable variables for a given alpha formulation, but isn't this approach also very vul…

I believe they may have used something on the lines of genetic-programming to create this equation - not sure about the high precision constants. The search space is compute intensive. Many years back, I used that technique to generate a profitable strategy. These things work and are different depending on the timeframe/sampling, stock, trend and money management.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#50

Post by twic »

traderthrow454 wrote: post #31
amiga_500 wrote: post #27

Earlier quoted context omitted.

I knew someone would come in with "liquidity". Many HFT jump out when things get volatile, when liquidity is actually required. Ultimately HFT is doing nothing of societal value, the race down to zero is never-ending and we are wasting huge amounts of resources on a totally pointless march towards zero. Exchanges should introduce random delays to allow market participants who really want to hedge / buy / sell, then w…

My initial comment was discussing speculative trading in general, but since you mostly brought up some common anti-HFT tropes I might as well address them. > Many HFT jump out when things get volatile, when liquidity is actually required. Do you have a citation on that? If you look the preliminary Q1 results of Virtu Financial [0] (only publicly traded HFT) they seem to be doing more trading than ever in these volati…

> If you look the preliminary Q1 results of Virtu Financial [0] (only publicly traded HFT) they seem to be doing more trading than ever in these volatile markets.

Similar story from Flow Traders:

https://www.flowtraders.com/sites/flow-traders/files/quarter...

Post reply on HN