Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Reinforcement Learning in the Algorithmic Trading Problem

arxiv.org

Post reply on HN
Print view View original

21–30 of 139 posts

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#21

Post by solidasparagus »

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

> "good" at trading don't publish papers

I think this is a little unfair. I've seen high-quality papers from phD students who then get hired by financial firms and were apparently very successful. Every good real-world AI system requires both good engineering and good science and it's disingenuous to suggest that all science that isn't actively being applied yet is BS.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#22

Post by nightski »

Qworg wrote: post #12
jawns wrote: post #5

Earlier quoted context omitted.

> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?

Trading is inherently zero sum.

That's only true in the sense of opportunity cost. I may buy something at $10 and sell it at $15 making a $5 profit. Then it may go to $20. Did I lose $5/share? Sure. But in reality I wasn't a "loser".

I find that in reality opportunity cost rarely matters.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#23

Post by awesome_dude »

jawns wrote: post #5
traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?

A famous example of a strategy that was published, and shared, whilst still profitable is that of Benjamin Graham, several of his students went on to be incredibly successful traders (we all know about Buffett, right?)

But that's an exception rather than a rule

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#24

Post by traderthrow454 »

amiga_500 wrote: post #11
jawns wrote: post #5

Earlier quoted context omitted.

> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?

> Or does it always reduce down to "I win only because you lose"? It is a zero sum game. Nobody is producing anything, therefore for one to win another must lose.

I think thats an overly simplistic view of things. The market is big and many participants trade at different frequencies. Large pension funds need liquidity to move big blocks of stock for their quarterly and monthly rebalances, and the big medium term statistical arbitrage traders provide liquidity for them to do so. HFT players provide liquidity for the stat arb players. The classes of participants with different frequencies actually help one another, while there is competition for alpha within strategies with similar holding periods. Overall the system creates an extremely efficient and liquid system for valuing and exchanging equity - the very system that empowers YCombinator and other Venture investors to make VC investments knowing that their winners will eventually IPO or be bought by public companies.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#25

Post by traK6Dcm »

solidasparagus wrote: post #21
traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

> "good" at trading don't publish papers I think this is a little unfair. I've seen high-quality papers from phD students who then get hired by financial firms and were apparently very successful. Every good real-world AI system requires both good engineering and good science and it's disingenuous to suggest that all science that isn't actively being applied yet is BS.

I don't claim that all the science that isn't actively being applied yet is BS, but this kind of science typically happens within trading firms, tested on real-world data, and is not being published on arXiv.

As a side note, what this specific paper here did is neither novel not innovative, so it's very fair to criticize it. A3C is 4 years old, and they just take it and run it on some data. It's like downloading a convnet and running it on MNIST. There have been hundreds of papers on RL + Trading. I see them in my arXiv emails every other day and they all do the same thing.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#26

Post by lordnacho »

Former hedge fund and HFT quant trader here. There's a lot of papers to be found claiming some sort of strategy. I don't want to go to cynicism immediately. But we'll get there:

- Trading isn't just about deciding what to buy and sell, the sexy part that everyone thinks is great. I even had colleagues who thought they were special because they worked closer to the strategies, which meant that certain less glamourous parts were neglected.

- Less glamourous parts like coding the software to read in the market data and send out orders.

- Less glamourous parts like schmoozing with brokers to get them to lower your costs.

- And maintaining infrastructure, which somehow people think should come as part of coding.

Now I'm not saying that RL won't help you. It's just that focusing on the "intelligent" part of the trading system tends to lead to disappointment, as you discover some unknown restrictions on your model that you hadn't thought of. Things like when you find out short selling was prohibited during the period that your model backtest was shorting.

My main red flags when reading papers are:

- Choosing a dataset from a small market. Basically any market that isn't the US or Western Europe large caps. You'll discover both price impact and high fees quite late in the game.

- Choosing a very small subset of the market. Smaller n, more noise and overfitting.

- Short periods. N again.

- Long intervals between decision making. N again again.

That's not to say there's nothing useful to be read though. You might be inspired by something you come across.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#27

Post by amiga_500 »

traderthrow454 wrote: post #24
amiga_500 wrote: post #11

Earlier quoted context omitted.

> Or does it always reduce down to "I win only because you lose"? It is a zero sum game. Nobody is producing anything, therefore for one to win another must lose.

I think thats an overly simplistic view of things. The market is big and many participants trade at different frequencies. Large pension funds need liquidity to move big blocks of stock for their quarterly and monthly rebalances, and the big medium term statistical arbitrage traders provide liquidity for them to do so. HFT players provide liquidity for the stat arb players. The classes of participants with different…

I knew someone would come in with "liquidity".

Many HFT jump out when things get volatile, when liquidity is actually required.

Ultimately HFT is doing nothing of societal value, the race down to zero is never-ending and we are wasting huge amounts of resources on a totally pointless march towards zero. Exchanges should introduce random delays to allow market participants who really want to hedge / buy / sell, then we can shift some of the resources to the real world. The costs required to compete at the lowest latencies are large, and forcing small/medium players out the game, as the investment cost is large, which is also bad.

The system is hugely inefficient. The costs as latencies get lower are ever higher, for an extremely similar end result. The law of diminishing returns.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#28

Post by currymj »

traK6Dcm wrote: post #19
deehouie wrote: post #14

Earlier quoted context omitted.

While it's so easy to dismiss someone's work as flawed (sure, backtest is illusional but do you have anything better?), which I think it may be, I always read it and try to understand what they're up to. Sure, academic folks may have no clue about market microstructure and other complexities, but if they could solve, or make some way toward solving the difficult problems in stochastic processes, they're already worth…

I actually believe that trading is an interesting problem that should be studied more in Academia and Machine Learning. It has many aspects (sparse rewards, long-time horizons, simulation-to-real-world transfer, non-stationary data distributions, etc) that current ML algorithms struggle with. Unfortunately it seem like most ML people are not really interested in trading, perhaps because it has such a bad reputation (…

i think it’s just because they want to publish, and as you say it’s not easy to find a really good publicly available dataset or simulator. I think if these were publicly available and there were a Python package, people would rapidly get interested in RL for trading. (if it even works for trading — i don’t know much about it but maybe simpler techniques work best, in which case there would be little chance of producing a publishable paper.)

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#29

Post by vecter »

Qworg wrote: post #12
jawns wrote: post #5

Earlier quoted context omitted.

> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?

Trading is inherently zero sum.

Zero-sum in wealth, but not zero-sum in utility. Otherwise, people wouldn't trade at all.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#30

Post by solidasparagus »

traK6Dcm wrote: post #25
solidasparagus wrote: post #21

Earlier quoted context omitted.

> "good" at trading don't publish papers I think this is a little unfair. I've seen high-quality papers from phD students who then get hired by financial firms and were apparently very successful. Every good real-world AI system requires both good engineering and good science and it's disingenuous to suggest that all science that isn't actively being applied yet is BS.

I don't claim that all the science that isn't actively being applied yet is BS, but this kind of science typically happens within trading firms, tested on real-world data, and is not being published on arXiv. As a side note, what this specific paper here did is neither novel not innovative, so it's very fair to criticize it. A3C is 4 years old, and they just take it and run it on some data. It's like downloading a co…

I was referring to these statements:

> This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system

> Whenever you see a paper ending with a backtest you can already assume it's BS.

Post reply on HN