Using Reinforcement Learning in the Algorithmic Trading Problem
1–10 of 139 posts
Re: Using Reinforcement Learning in the Algorithmic Trading Problem
#2As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system.
Not only is it (very!) easy to overfit backtests (especially with so little data they are using here), but backtests are nothing like the real world. In the real world there are HFT traders front-running you, latency, jitter, fees, hidden order types, slippage, and a lot of other complexities that don't fit into a short HN post. Whenever you see a paper ending with a backtest you can already assume it's BS.
It's similar to training a robot in an extremely simplified 2D simulation environment without physics or other interactions, and then claiming one has built a real robot. A mistake many people make is believing that trading is all about AI. But in reality, the model often matters less than infrastructure/latency/system/data issues.
In addition to that, people who are actually "good" at trading don't publish papers, they silently make money. Papers are typically published by academics or students who have never built anything profitable but would like to put a paper on their resume. I have yet to see a single good academic paper about trading.
[0] https://www.tradientblog.com/2019/11/lessons-learned-buildin...
Re: Using Reinforcement Learning in the Algorithmic Trading Problem
#3It looks like their code is assuming a fixed commission of 20 rubles, which is apparently equivalent to US$0.27. Does anyone know if that's realistic? https://github.com/evgps/a3c_trading/blob/master/configs.py#...
Depending on how liquid the market they studied is, code that assumes there is never any slippage may not be very realistic.
There's no comparison to a simple buy-and-hold strategy, which may be less interesting from a computer science perspective but is a good way to avoid spending lots of money on transaction costs.
(I once posted my own algorithmic trading project to HN that had very flawed, naive assumptions about what trades could be executed.)
Re: Using Reinforcement Learning in the Algorithmic Trading Problem
#4Every time such a paper like this comes out, I have to ask myself: if they knew how to make money like this, why would they tell anyone?
Re: Using Reinforcement Learning in the Algorithmic Trading Problem
#5As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…
> people who are actually "good" at trading don't publish papers, they silently make money
I've long understood that this was true. It makes intuitive sense.
But are there any cases where it is not true?
Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor?
Or does it always reduce down to "I win only because you lose"?
Re: Using Reinforcement Learning in the Algorithmic Trading Problem
#6As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…
This is a really valuable opinion, especially because it's easy to be misled about these things for those of us who are novices in HFT.
Re: Using Reinforcement Learning in the Algorithmic Trading Problem
#7As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…
I share your skepticism.
Now, since you appear to know about these things, among all the available papers/article/blogposts/books is there any that you would recommend as being less wrong than the rest? For example, a while ago I read this book [1], and it didn't seem so bad, but I'm not in the industry. Can you recommend anything, even with caveats?
[1] https://www.amazon.com/gp/product/B00BZ9WAVW/ref=dbs_a_def_r...
Re: Using Reinforcement Learning in the Algorithmic Trading Problem
#8As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…
++ this.
If they haven't tested this in actual trades and measured results, it's probably worthless. Even backtested strategies at actual firms observe decays (or don't work) when they get put live. And those are places where they invest in (and are incentivized to get right!) backtesting methodology.
Re: Using Reinforcement Learning in the Algorithmic Trading Problem
#9As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…
> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?
I don't think it necessarily has to be true. I also built a profitable system and wrote about it, but I didn't share all the details. Not even close. There are just too many small details that must be "just right" that they would fill a whole book. It's kind of like building an operating system from scratch. It's not something you can put into a single post or paper. There isn't one "trick" that suddenly makes it all profitable - it's a combination of so many small details.
Then there is the cultural aspect. People who are working in trading are just not used to sharing openly. They don't write online, or anywhere. They are not even allowed to write due to their employers. And people who work in academia are naturally not working on "production" systems - their only job is to write, not to build. So you almost never see people in the intersection of: writes-online & understands-trading & is-not-in-academia
Re: Using Reinforcement Learning in the Algorithmic Trading Problem
#10Every time such a paper like this comes out, I have to ask myself: if they knew how to make money like this, why would they tell anyone?
It's a weird subject for academic study in the first place. Trying do do trading strategies that beat the market is one of the few things that really is a zero-sum game. In the absence of independent scientific interest to optimizing these strategies, what's the point? You might as well study how to optimize strategies for ultimate frisbee or something.