Using Reinforcement Learning in the Algorithmic Trading Problem • Hacker News Board

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#121

Post by Traster » Thu, Apr 30, 2020, 12:18 PM UTC

derriz wrote: post #116

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

I also worked in HFT and have no idea what you mean when you say other HFT shops can front-run your orders? To front run someone’s order you need to have advance information of their order s? Normally this means the front runner is operating as a broker. I can’t imagine any HFTs using other HFTs as brokers to forward their orders to the exchange?

The distinction is between algorithmic traders - and HFT. HFT traders often find ways of making money from algorithmic traders. Especially if the algorthmic traders are doing things like VWAP.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#122

Post by mD5pPxMcS6fVWKE » Thu, Apr 30, 2020, 12:28 PM UTC

Problem is, there is competition, so any "feature" your ML has discovered, will fade away as other MLs discover it too. So it has a tendency to become a completely featureless stochastic system. Temporary features can exist though.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#123

Post by derriz » Thu, Apr 30, 2020, 12:48 PM UTC

Traster wrote: post #121

derriz wrote: post #116

Earlier quoted context omitted.

I also worked in HFT and have no idea what you mean when you say other HFT shops can front-run your orders? To front run someone’s order you need to have advance information of their order s? Normally this means the front runner is operating as a broker. I can’t imagine any HFTs using other HFTs as brokers to forward their orders to the exchange?

The distinction is between algorithmic traders - and HFT. HFT traders often find ways of making money from algorithmic traders. Especially if the algorthmic traders are doing things like VWAP.

I don't understand your point or how it explains how HFT companies can "front-run" other HFT companies?

Front-running is when someone with a fiduciary duty - typically a broker or dealer - takes an order from a client and then trades on their own book BEFORE executing the client's order knowing the effect of the clients order on the market and knowing that they can exploit this effect for their own benefit.

I know of no HFTs which have such a relationship with rival HFTs and can't even imagine such a relationship existing never mind it being a frequent cause of why strategies perform poorly for HFTs.

Front-running hasn't been a feature of markets for decades at least. Any sniff of front-running would have the SEC or CFTC fine your company into oblivion and possibly result in jail time or at the very least lifetime bans from the financial industry.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#124

Post by WJW » Thu, Apr 30, 2020, 1:16 PM UTC

anjc wrote: post #104

vecter wrote: post #97

Earlier quoted context omitted.

That's a great way to go broke very quickly.

Only if you don't have infinite money on the side, in which case it's guaranteed to be profitable

You also need the other side of your trades to have infinite money. But in any case, why are you trading at all if you already have infinite money?

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#125

Post by outlace » Thu, Apr 30, 2020, 2:49 PM UTC

jmeister wrote: post #102

outlace wrote: post #43

Anyone interested in delving into reinforcement learning should check out our book https://www.manning.com/books/deep-reinforcement-learning-in... - we cover actor critic (used in this paper) to multi-agent to relational/attention models

Thanks, looks very useful. Since you seem to be industry practitioners: I moved away from RL 10 yrs back disillusioned with lack of real-world applicability. Has that changed significantly? The only major name I’ve heard of is Vowpal Wabbit. Maybe there are more applications being done in stealth. Any insight? Thanks

Honestly, not much has changed. The primary use cases continue to be things like robotics, playing games, finance etc. I’m interested in RL academically from a computational neuroscience standpoint (using RL to model cognition) but also as applied toward healthcare problems.

However, I don’t think the current limited use of RL is a permanent situation just that the most exciting uses of RL are extremely difficult problems that involve long-time horizons and planning. For example, RL could be used to automatically prove mathematical theorems which would be amazing. But it’s a really hard problem for various reasons. Still a lot of progress to be made.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#126

Post by machinehermit » Thu, Apr 30, 2020, 4:00 PM UTC

scottlocklin wrote: post #73

traK6Dcm wrote: post #48

Earlier quoted context omitted.

You make a good point. I've also gotten into trading because I enjoy the algorithmic and mathematical aspects, and I would love to share more of what has been working for me and write extensively about it. And there are probably more people like that out there. However, trading has such a bad reputation and uncertain future that I am not sure that's a good career move. I'm torn. You're right that there are probably s…

>I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post. You should really google up something called the Gell-Mann amnesia effect. 99.9% of everything is shit. Including biology, CS, ML and especially "AI." Of course trading papers are even more universally shit, but once in a while someone publis…

Thanks for the Gell-Mann amnesia effect.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#127

Post by machinehermit » Thu, Apr 30, 2020, 4:11 PM UTC

jawns wrote: post #5

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?

It is an absurdity to believe there has never been a good trading strategy published in a paper.

The real value though of publishing a trading strategy is in signaling to future employers.

Ultimately, money is made by the ability to come up with new strategies. Any single strategy is only going to live for so long before it dies and it is no longer profitable.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#128

Post by webel0 » Thu, Apr 30, 2020, 4:18 PM UTC

derriz wrote: post #116

traK6Dcm wrote: post #2

As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are…

I also worked in HFT and have no idea what you mean when you say other HFT shops can front-run your orders? To front run someone’s order you need to have advance information of their order s? Normally this means the front runner is operating as a broker. I can’t imagine any HFTs using other HFTs as brokers to forward their orders to the exchange?

If someone starts running this model couldn’t you just run the model yourself to predict the order?

I’m sure that there is market microstructure stuff/front running practicalities that would make this harder than it sounds but still you wouldn’t completely be in the dark.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#129

Post by derriz » Thu, Apr 30, 2020, 5:30 PM UTC

webel0 wrote: post #128

derriz wrote: post #116

Earlier quoted context omitted.

I also worked in HFT and have no idea what you mean when you say other HFT shops can front-run your orders? To front run someone’s order you need to have advance information of their order s? Normally this means the front runner is operating as a broker. I can’t imagine any HFTs using other HFTs as brokers to forward their orders to the exchange?

If someone starts running this model couldn’t you just run the model yourself to predict the order? I’m sure that there is market microstructure stuff/front running practicalities that would make this harder than it sounds but still you wouldn’t completely be in the dark.

But how would doing that constitute front running?

Front running has a specific definition in terms of market regulation (see my other comment) and what you describe is not front-running.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#130

Post by p1esk » Thu, Apr 30, 2020, 6:22 PM UTC

jerzyt wrote: post #119

p1esk wrote: post #90

Earlier quoted context omitted.

I don't get it - if you have accurate historical data, how is this different from having access to current real-time data? Why can't you pretend you live 20 years in the past and use the data you have as if it were real-time?

Fundamentally, the issue is that in real time you may not be able to make the trade that your algorithm chose. You could get close if you had the actual book prices at any given time, but even then, you might lose to someone who is 1 millisecond faster. So no, backtesting can simulate reality. Interactive Brokers offers a simulated account where you can practice "live" trading, although it's still not the same, since…

Good point, but I was mainly thinking about making a single good decision to multiply your investment, not HFT. Like identifying that it was a good idea to invest in Tesla stock 7 years ago.