Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Reinforcement Learning in the Algorithmic Trading Problem

arxiv.org

Post reply on HN
Print view View original

71–80 of 139 posts

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#71

Post by IMAYousaf »

phobar wrote: post #66
nycdatasci wrote: post #57

All profitable automated trading strategies that I'm aware of target a specific inefficiency in the market. What is the inefficiency here? If you can't articulate the inefficiency, it's probably best not to employ the strategy.

Interesting, could you describe one inefficiency that was exploited in the past? I could imagine buying/selling due to spreads between exchanges but is there another not as obvious example?

I can describe one inefficiency from sports gambling. There is a famous NBA Gambler named Haralabos Voulgaris. He realized that the points total prediction for a game, let's say 100 points scored for Team A, was merely sliced in half to represent the half-time score. However, the pace of the first half is markedly different from the pace of the second half, thus points are scored at an uneven clip. He exploited that inefficiency for a while to great success.

Like sports gambling, a lot of the financial products we trade are obviously built by humans using rules, and arbitraging the intrinsic rules and regulations around said products. Think about Forex trading where you convert currency into currency. One of the key strategies is to find and identify brief negative cycles, for example, in the hope that converting US Dollars to Euros to Yen back to US Dollars leaves you with more dollars than you started out with.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#72

Post by nuclearnice1 »

buzzkillington wrote: post #56
vecter wrote: post #29

Earlier quoted context omitted.

Zero-sum in wealth, but not zero-sum in utility. Otherwise, people wouldn't trade at all.

That's not true, it just needs to cost you more to not play than it does to play. Something like: | | Play | Don't Play | |------------+------+------------| | Play | -5 | +10 | | Don't Play | -10 | 0 |

Your payoffs are not zero sum.

> In game theory and economic theory, a zero-sum game is a mathematical representation of a situation in which each participant's gain or loss of utility is exactly balanced by the losses or gains of the utility of the other participants.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#73

Post by scottlocklin »

traK6Dcm wrote: post #48
smdz wrote: post #46

Earlier quoted context omitted.

> In addition to that, people who are actually "good" at trading don't publish papers, they silently make money. Well, that is mostly true. But never discount anything. There are people like me who used to love the data analysis and prediction part in these markets. I got hooked to the markets because of it. I was not interested in making money and naively thought my average pay was good enough. When I first built (o…

You make a good point. I've also gotten into trading because I enjoy the algorithmic and mathematical aspects, and I would love to share more of what has been working for me and write extensively about it. And there are probably more people like that out there. However, trading has such a bad reputation and uncertain future that I am not sure that's a good career move. I'm torn. You're right that there are probably s…

>I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post.

You should really google up something called the Gell-Mann amnesia effect. 99.9% of everything is shit. Including biology, CS, ML and especially "AI."

Of course trading papers are even more universally shit, but once in a while someone publishes a non obvious to me risk factor.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#74

Post by fnbr »

MasterScrat wrote: post #51
fnbr wrote: post #15

An additional problem with this is that they use A3C here for trading. A3C is known to not be suitable for adversarial environments (e.g. board games, like Chess). I wrote a paper that demonstrated that A3C is as exploitable as a uniform random strategy in board games (specifically, some poker variants): https://arxiv.org/abs/2004.09677 (Exploitable is a technical term that is defined in the paper; basically, it's "h…

> A3C is known to not be suitable for adversarial environments Interesting! What are the main papers in this area? Any intuition why this is the case? is it because A2C generally results in brittle policies?

It’s mostly an issue that A2C isn’t designed for adversarial environments. It also doesn’t have any notion of hidden information, while other algorithms (eg CFR) explicitly handle this. There’s a well-known phenomena of cycling, where agent A will beat agent B which beats agent C which beats agent A; A2C can exhibit this. Think of rock/paper/scissors- AlwaysRock beats AlwaysScissors which beats AlwaysPaper. To avoid this, you typically need to do some sort of averaging.

The alphastar paper and blog post do a good job discussing these issues as they had similar problems. I’d say that’s a great starting point (and then following their references).

Blog post:

https://deepmind.com/blog/article/alphastar-mastering-real-t...

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#75

Post by vmception »

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

> Not only is it (very!) easy to overfit backtests (especially with so little data they are using here), but backtests are nothing like the real world.

I know this, and I ran a company where people should know this, but so many people are so easily swayed by "authority"

like, so and so made trading programs for Investment Bank Co 20 years ago so you know their trading algorithm has to have merit

uh no, they are not retired, they are broke and can't even fund $10k into a trading account to try it

at this point all I would say is just smile and nod.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#76

Post by ivalm »

yorwba wrote: post #53
nightski wrote: post #22

Earlier quoted context omitted.

That's only true in the sense of opportunity cost. I may buy something at $10 and sell it at $15 making a $5 profit. Then it may go to $20. Did I lose $5/share? Sure. But in reality I wasn't a "loser". I find that in reality opportunity cost rarely matters.

If you buy something at $10 and sell it at $15, where are the $5 profit coming from? From the other market participants, e.g. someone selling to you for $10 and later buying it back for $15, losing $5 in the process. Your profit and their loss sum to zero, which is what "zero-sum" means. It has absolutely nothing to do with opportunity cost, or whether you, personally, are a "loser". But if you're a "winner", someone…

Asset values can just increase. Alice who has $10 and 0 units buys 10 units from Bob who has 10 units and $0 dollars. Alice then sell back to Bob 5 units for $10 dollars. Alice now has 5 units (worth $10) and $10, Bob now has 5 units (worth $10). Total wealth in the system went from $20 to $30.

In addition, companies produce things, some of that wealth gets returned to the investors through dividends, interest (eg on bonds) and buy backs (in my example, let's say each unit generates $1 in dividend, now total wealth is $40(!) while starting at $20, including $20 cash (starting from $10) and $20 worth of units (starting from $10)).

In fact, we see this growth everywhere around us as both the amount of people and the amount of goods and services per person is increasing!

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#77

Post by buzzkillington »

nuclearnice1 wrote: post #72
buzzkillington wrote: post #56

Earlier quoted context omitted.

That's not true, it just needs to cost you more to not play than it does to play. Something like: | | Play | Don't Play | |------------+------+------------| | Play | -5 | +10 | | Don't Play | -10 | 0 |

Your payoffs are not zero sum. > In game theory and economic theory, a zero-sum game is a mathematical representation of a situation in which each participant's gain or loss of utility is exactly balanced by the losses or gains of the utility of the other participants.

Yes? The point is that no one playing has the highest payout for the group[1].

[1] You can change the +10 to +9 if you want to make it the absolute highest total payout.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#78

Post by chnsh »

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

I have your post saved and have gone through it many times, thanks for writing it - big fan!

As a student who is looking to get started with trading and enjoys the mathematical/analysis part of it, do you have advice of where to begin? I find very few resources in this area and its very hard to get on this career path - my experience is on the ML side if things and I want to transition into trading. Any advice will be really helpful - thanks!

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#79

Post by godelmachine »

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

At the risk of digressing, might I ask if the dough to be minted is good in Quant/ HFT / Algorithmic trading?

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#80

Post by lumost »

p1esk wrote: post #68
traK6Dcm wrote: post #48

Earlier quoted context omitted.

You make a good point. I've also gotten into trading because I enjoy the algorithmic and mathematical aspects, and I would love to share more of what has been working for me and write extensively about it. And there are probably more people like that out there. However, trading has such a bad reputation and uncertain future that I am not sure that's a good career move. I'm torn. You're right that there are probably s…

So how should we evaluate the quality of a paper on trading AI? I mean the authors might not have access to real data, but their ideas might still be good.

There are some ML problems where it is fundamentally impossible to use historical data to make accurate forward looking predictions as its not IID. These fields require you very carefully capture data on sub-optimal choices. In the case of trading this means making explicitly bad trading decisions some portion of the time, and teams that have done this at any scale are unlikely to share the data.

In the case of trading, any paper not tackling these issues head on is not likely to be useful.

Post reply on HN