Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Reinforcement Learning in the Algorithmic Trading Problem

arxiv.org

Post reply on HN
Print view View original

11–20 of 139 posts

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#11

Post by amiga_500 »

jawns wrote: post #5
traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?

> Or does it always reduce down to "I win only because you lose"?

It is a zero sum game. Nobody is producing anything, therefore for one to win another must lose.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#12

Post by Qworg »

jawns wrote: post #5
traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?

Trading is inherently zero sum.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#13

Post by nine_k »

jawns wrote: post #5
traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?

To spread one's wealth, one can donate to charities.

Opening up one's secrets of trading seems to only make sense if one has found deeper, more effective secrets, so that the old crop is not going to be seriously competitive, but a bit of good PR would come in handy.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#14

Post by deehouie »

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

While it's so easy to dismiss someone's work as flawed (sure, backtest is illusional but do you have anything better?), which I think it may be, I always read it and try to understand what they're up to. Sure, academic folks may have no clue about market microstructure and other complexities, but if they could solve, or make some way toward solving the difficult problems in stochastic processes, they're already worth my effort.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#15

Post by fnbr »

An additional problem with this is that they use A3C here for trading. A3C is known to not be suitable for adversarial environments (e.g. board games, like Chess).

I wrote a paper that demonstrated that A3C is as exploitable as a uniform random strategy in board games (specifically, some poker variants): https://arxiv.org/abs/2004.09677

(Exploitable is a technical term that is defined in the paper; basically, it's "how much can someone who knows everything about your strategy beat you by?")

So I would be very surprised if this survives contact with other traders.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#16

Post by traK6Dcm »

credit_guy wrote: post #7
traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

I share your skepticism. Now, since you appear to know about these things, among all the available papers/article/blogposts/books is there any that you would recommend as being less wrong than the rest? For example, a while ago I read this book [1], and it didn't seem so bad, but I'm not in the industry. Can you recommend anything, even with caveats? [1] https://www.amazon.com/gp/product/B00BZ9WAVW/ref=dbs_a_def_r...

In general, books are a much better source of information than papers or blog posts when it comes to trading. I haven't read the one you posted, but a few I can recommend:

[0] is okay. I disagree with a lot in there, but it's pretty well written and one of the better books on the subject. [1] Is very old, but it's one of my favorites. It's very mathematical. The ideas still apply today. [2] Is a good introduction overview

[0] https://www.amazon.com/Advances-Financial-Machine-Learning-M...

[1] https://www.amazon.com/Introduction-High-Frequency-Finance-R...

[2] https://www.amazon.com/Trading-Exchanges-Microstructure-Prac...

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#17

Post by solotronics »

amiga_500 wrote: post #11
jawns wrote: post #5

Earlier quoted context omitted.

> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?

> Or does it always reduce down to "I win only because you lose"? It is a zero sum game. Nobody is producing anything, therefore for one to win another must lose.

this would only be true if it was a closed system. the central banks essentially magic money into existence and put it into the market through convoluted methods.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#18

Post by whymauri »

solotronics wrote: post #17
amiga_500 wrote: post #11

Earlier quoted context omitted.

> Or does it always reduce down to "I win only because you lose"? It is a zero sum game. Nobody is producing anything, therefore for one to win another must lose.

this would only be true if it was a closed system. the central banks essentially magic money into existence and put it into the market through convoluted methods.

My understanding is that while all markets are not zero-sum, that high-frequency trade amongst trading firms approaches zero-sum.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#19

Post by traK6Dcm »

deehouie wrote: post #14
traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

While it's so easy to dismiss someone's work as flawed (sure, backtest is illusional but do you have anything better?), which I think it may be, I always read it and try to understand what they're up to. Sure, academic folks may have no clue about market microstructure and other complexities, but if they could solve, or make some way toward solving the difficult problems in stochastic processes, they're already worth…

I actually believe that trading is an interesting problem that should be studied more in Academia and Machine Learning. It has many aspects (sparse rewards, long-time horizons, simulation-to-real-world transfer, non-stationary data distributions, etc) that current ML algorithms struggle with.

Unfortunately it seem like most ML people are not really interested in trading, perhaps because it has such a bad reputation (which is IMO unjustified) - so they work on games instead :)

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#20

Post by traderthrow454 »

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

If you want to read useful academic papers about trading there is one author in particular who is actually not bad - Zura Kakushadze. Most of his stuff is applicable to mid-frequency trading, not HFT. He worked at WorldQuant (reputable trading firm) and the founder of WQ, Igor Tulchinsky, is a coauthor on one of his papers.

Example of a pretty interesting and accessible one - is "101 Formulaic Alphas" [0].

[0] - https://arxiv.org/pdf/1601.00991.pdf

Post reply on HN