Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Reinforcement Learning in the Algorithmic Trading Problem

arxiv.org

Post reply on HN
Print view View original

131–139 of 139 posts

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#131

Post by usefulcat »

yorwba wrote: post #53
nightski wrote: post #22

Earlier quoted context omitted.

That's only true in the sense of opportunity cost. I may buy something at $10 and sell it at $15 making a $5 profit. Then it may go to $20. Did I lose $5/share? Sure. But in reality I wasn't a "loser". I find that in reality opportunity cost rarely matters.

If you buy something at $10 and sell it at $15, where are the $5 profit coming from? From the other market participants, e.g. someone selling to you for $10 and later buying it back for $15, losing $5 in the process. Your profit and their loss sum to zero, which is what "zero-sum" means. It has absolutely nothing to do with opportunity cost, or whether you, personally, are a "loser". But if you're a "winner", someone…

Market makers (market participants who are interested in either buying or selling) are like used car dealers. If you are in the market to buy or sell a car, you could choose to find a buyer or seller yourself, and you may well get a better price going that route. It will also usually require more work from you and take longer than if you just went to a car dealer. So that's the tradeoff: save time via an intermediary (who will likely profit from the transaction), or do more work yourself and possibly get a better price.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#132

Post by jmeister »

johnmoberg wrote: post #110
jmeister wrote: post #102

Earlier quoted context omitted.

Thanks, looks very useful. Since you seem to be industry practitioners: I moved away from RL 10 yrs back disillusioned with lack of real-world applicability. Has that changed significantly? The only major name I’ve heard of is Vowpal Wabbit. Maybe there are more applications being done in stealth. Any insight? Thanks

You might be interested in the recently-launched Covariant ( https://covariant.ai/ ), they apparently actually have systems in production. Pieter Abbeel is one of the founders and they have some pretty "heavy" investors, like Jeff Dean, Geoffrey Hinton, and Yann LeCun.

Thanks! OpenAI has made some progress on this problem too: https://news.ycombinator.com/item?id=21259765

Looks like these two places are on the cusp of a major breakthrough in RL/robotics!

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#133

Post by iliicit »

i've been trading crypto in large volumes at high frequencies for quite some time now. my models were plain as yogurt feed-forward neural nets. i would engineer some dumb features, sample the at random data, assign the labels (that translate into trading decisions), and train the model. then push to prod, sit back, and relax while the balance grows like a mushroom cloud. just kidding, before that i would grow gray hair while backtesting, debugging issues, etc.

one of the hard problems was labeling the data. knowing that the price is going up 10 bps one minute from now, should i buy? maybe. but what if it's going to crash 100 bps right after this? probably should sell instead.

reinforcement learning promises to eliminate the need to assign labels in the training data. the agent will try a bunch of different variants at random and eventually will choose the most optimal one knowing the state of the working, i.e. the state of the markets. at training time i only need to feed it the features data. another benefit is that backtesting and model training is sort of fused into a single process. rl model is optimizing pnl, and not the label classification score (as in the nn model). with proper train-test-validation split, the most performant rl model can go straight into production (helping me to keep some of my hair brown)

while all the bits and pieces seem straightforward i never managed to tune rl model to work better in the backtest compared to the good old old nn models. maybe i have never been closer to the gold vein, but for now, i abandoned my efforts to build a performant rl agent if favor of nn models.

amen.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#134

Post by formalsystem »

anjc wrote: post #108
formalsystem wrote: post #94

So many papers by "smart" people predicting the past

How do you propose that anyone evaluates their algorithm on the future?

start a fund with your ML approach - evaluate how much money you've made after 1m,6m,1y,2y,10y etc..

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#135

Post by 3pt14159 »

Qworg wrote: post #12
jawns wrote: post #5

Earlier quoted context omitted.

> people who are actually "good" at trading don't publish papers, they silently make money I've long understood that this was true. It makes intuitive sense. But are there any cases where it is not true? Is it possible to "spread the wealth" when it comes to trading, or any money-making endeavor? Or does it always reduce down to "I win only because you lose"?

Trading is inherently zero sum.

Not always. People have different time horizons on the utility of money and non-linear outcomes on risk.

For example, a gold miner may sell gold futures to guarantee that he won't go out of business once the construction of the new gold mine is complete. There are many other examples.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#136

Post by Traster »

derriz wrote: post #123
Traster wrote: post #121

Earlier quoted context omitted.

The distinction is between algorithmic traders - and HFT. HFT traders often find ways of making money from algorithmic traders. Especially if the algorthmic traders are doing things like VWAP.

I don't understand your point or how it explains how HFT companies can "front-run" other HFT companies? Front-running is when someone with a fiduciary duty - typically a broker or dealer - takes an order from a client and then trades on their own book BEFORE executing the client's order knowing the effect of the clients order on the market and knowing that they can exploit this effect for their own benefit. I know of…

Ok, well let's say you're using an algorithm to trade, and an HFT firm identifies what your 'algorithm' is doing, they're going to front run you - whether that be using VWAP or flashing 10 lots every 30 seconds. And both of those absolutlely happen. They're not going to literally 'know' what you're going to do, but some algos are pretty obvious and somewhat exploitable.

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#137

Post by deehouie »

traK6Dcm wrote: post #100
deehouie wrote: post #86

Earlier quoted context omitted.

Ther are plenty of free datasets out there. You can get upward of 10 yrs of daily OHLC stock data on yahoo finance. The amazing thing is yf has S&P 500 index since 1927. Free! Quandl has many free, or low cost stock market/commodity datasets. I'm not sure what you mean by a "simulator". One of the greatest challenge applying RL to stock mkt is precisely that the market itself is not a MDP.

I don't think daily OHLCV data is a good data source. First of all, it's too little because of the data distribution shift over time. It's also driven significantly by outliers and events outside of the data (news, etc). There's way too much noise in daily prices that most of the signal is drowned out (longer time horizons = more uncertainty). I don't believe you can find any edge looking at daily data. This kind of…

What you said about daily data is precisely what makes stock mkt so interesting and challenging : nonstationarity.

"outliers and events outside of the data, news" : these are precisely the stuff your models need to learn, and the fact that you consider them noise tells me most folks have no clue how to predict these "noise".

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#138

Post by derriz »

Traster wrote: post #136
derriz wrote: post #123

Earlier quoted context omitted.

I don't understand your point or how it explains how HFT companies can "front-run" other HFT companies? Front-running is when someone with a fiduciary duty - typically a broker or dealer - takes an order from a client and then trades on their own book BEFORE executing the client's order knowing the effect of the clients order on the market and knowing that they can exploit this effect for their own benefit. I know of…

Ok, well let's say you're using an algorithm to trade, and an HFT firm identifies what your 'algorithm' is doing, they're going to front run you - whether that be using VWAP or flashing 10 lots every 30 seconds. And both of those absolutlely happen. They're not going to literally 'know' what you're going to do, but some algos are pretty obvious and somewhat exploitable.

That's not front running. You can only front run and order if you're handling the order for a third party.

Being faster than someone else isn't "front running" them nor is spotting patterns in other participant's behaviour and exploiting those patterns. The definition of "front running" is reasonably specific: https://en.wikipedia.org/wiki/Front_running

Re: Using Reinforcement Learning in the Algorithmic Trading Problem

#139

Post by nuclearnice1 »

buzzkillington wrote: post #77
nuclearnice1 wrote: post #72

Earlier quoted context omitted.

Your payoffs are not zero sum. > In game theory and economic theory, a zero-sum game is a mathematical representation of a situation in which each participant's gain or loss of utility is exactly balanced by the losses or gains of the utility of the other participants.

Yes? The point is that no one playing has the highest payout for the group[1]. [1] You can change the +10 to +9 if you want to make it the absolute highest total payout.

Sorry I don’t follow. Can you elaborate?
Post reply on HN