Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We gave 5 LLMs $100K to trade stocks for 8 months

aitradearena.com

Post reply on HN
Print view View original

261–270 of 319 posts

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#261

Post by petercooper »

culi wrote: post #125
bcrosby95 wrote: post #12

> Grok ended up performing the best while DeepSeek came close to second. Almost all the models had a tech-heavy portfolio which led them to do well. Gemini ended up in last place since it was the only one that had a large portfolio of non-tech stocks. I'm not an investor or researcher, but this triggers my spidey sense... it seems to imply they aren't measuring what they think they are.

I'd like to see this study replicated during a bear market

Agreed. While I don’t see it outperforming long held funds, it’d be interesting to see if they could pick up on negative signals in the news feed, and also any potential advantage of not being emotional about its decisions.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#262

Post by Guillaume86 »

hobobaggins wrote: post #130
seanmcdirmid wrote: post #111

Earlier quoted context omitted.

We had this discussion in previous posts about congressional leaders who had the risk appetite to go tech heavy and therefore outperformed normal congress critters. Going heavy on tech can be rewarding, but you are taking on more risk of losing big in a tech crash. We all know that, and if you don't have that money to play riskier moves, its not really a move you can take. Long term it is less of a win if a tech bubb…

They didn't just outperform "normal" congress critters.. they also outperformed nearly every hedge fund on the planet. But they (meaning, of course, just one person and their spouse) are obviously geniuses.

They also outperformed themselves before being in a leader position...

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#263

Post by blitzar »

refactor_master wrote: post #139
tclancy wrote: post #118

Earlier quoted context omitted.

I mean, run the experiment during a different trend in the market and the results would probably be wildly different. This feels like chartists [1] but lazier. [1] https://www.investopedia.com/terms/c/chartist.asp

If you've ever read a blog on trading when LSTMs came out, you'd have seen all sorts of weird stuff with predicting the price at t+1 on a very bad train/test split, where the author would usually say "it predicts t+1 with 99% accuracy compared to t", and the graph would be an exact copy with a t+1 offset. So eye-balling the graph looks great, almost perfect even, until you realize that in real-time the model would've…

if you feed in price i.e. 280.1, 281.5, 281.9 ... you are going to get some pretty good looking results when it comes to predicting the next days price (t+1) with a margin of +/- a percent or so.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#264

Post by solotronics »

hxtk wrote: post #175
tshaddox wrote: post #67

Earlier quoted context omitted.

> It would almost be more interesting to specifically train the model on half the available market data, then test it on another half. Yes, ideally you’d have a model trained only on data up to some date, say January 1, 2010, and then start running the agents in a simulation where you give them each day’s new data (news, stock prices, etc.) one day at a time.

I suspect trading firms have already done this to the maximum extent that it's profitable to do so. I think if you were to integrate LLMs into a trading algorithm, you would need to incorporate more than just signals from the market itself. For example, I hazard a guess you could outperform a model that operates purely on market data with a model that also includes a vector embedding of a selection of key social and…

The part people are missing here is that if the trading firms are all doing something, that in itself influences the market.

If they are all giving the LLMs money to invest and the AIs generally buy the same group of stocks, those stocks will go up. As more people attempt the strategy it infuses fresh capital and more importantly signaling to the trading firms there are inflows to these stocks. I think its probably a reflexive loop at this point.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#265

Post by ekianjo »

olliepro wrote: post #19
bcrosby95 wrote: post #12

> Grok ended up performing the best while DeepSeek came close to second. Almost all the models had a tech-heavy portfolio which led them to do well. Gemini ended up in last place since it was the only one that had a large portfolio of non-tech stocks. I'm not an investor or researcher, but this triggers my spidey sense... it seems to imply they aren't measuring what they think they are.

A more sound approach would have been to do a monte carlo simulation where you have 100 portfolios of each model and look at average performance.

indeed, and also a "model" does not mean anything per se, you have hundreds of different prompts, you can layer agents on top, you can use temperature that will lead to different outcomes. The number of dimensions to explore is huge.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#266

Post by beezle »

cheeseblubber wrote: post #39

OP here. We realized there are a ton of limitations with backtest and paper money but still wanted to do this experiment and share the results. By no means is this statistically significant on whether or not these models can beat the market in the long term. But wanted to give everyone a way to see how these models think about and interact with the financial markets.

What were the risk adjusted returns? Without knowing that, this is all kind of meaningless. Being high beta in a rising market doesn't equate to anything brilliant.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#267

Post by stonemetal12 »

IgorPartola wrote: post #34
bcrosby95 wrote: post #12

> Grok ended up performing the best while DeepSeek came close to second. Almost all the models had a tech-heavy portfolio which led them to do well. Gemini ended up in last place since it was the only one that had a large portfolio of non-tech stocks. I'm not an investor or researcher, but this triggers my spidey sense... it seems to imply they aren't measuring what they think they are.

Yeah I mean if you generally believe the tech sector is going to do well because it has been doing well you will beat the overall market. The problem is that you don’t know if and when there might be a correction. But since there is this one segment of the overall market that has this steady upwards trend and it hasn’t had a large crash, then yeah any pattern seeking system will identify “hey this line keeps going up…

Would that work for LLMs though? They hypothetically trained on news papers from the second half of the data so they have knowledge of "future" events.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#268

Post by intalentive »

stouset wrote: post #153
chris_st wrote: post #110

Earlier quoted context omitted.

> you will have trained your model on market patterns that might not be in place anymore My working definition of technical analysis [0] [0]: https://en.wikipedia.org/wiki/Technical_analysis

I am frankly astonished at the number of otherwise-intelligent people who actually seem to believe in this stuff. One of the worst possible things to do in a competitive market is to trade by some publicly-available formulaic strategy. It’s like announcing your rock-paper-scissors move to your opponent in advance.

Technical analysis is a basket of heuristics. Support / resistance / breakout (especially around whole numbers) seems to reflect persistent behavior rooted in human psychology. Look at the heavy buying at the $30 mark here, putting a floor under silver: https://finviz.com/futures_charts.ashx?p=d&t=SI This is a common pattern it can be useful to know.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#269

Post by kqr »

Extremely similar earlier submission but focused on cryptocurrencies, using real money, and in real time: https://news.ycombinator.com/item?id=45976832

I'm extremely skeptical of any attempt to prevent leakage of future results to LLMs evaluated on backtesting. Both because this has beet shown in the literature to be difficult, and because I personally found it very difficult when working with LLMs for forecasting.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#270

Post by acrooks »

naet wrote: post #62

I used to work for a brokerage API geared at algorithmic traders and in my experience anecdotal experience many strategies seem to work well when back-tested on paper but for various reasons can end up flopping when actually executed in the real market. Even testing a strategy in real time paper trading can end up differently than testing on the actual market where other parties are also viewing your trades and makin…

A really important part of this is the emotional component. When real money is involved, then you will sometimes face actual losses. It’s hard for a human to completely trust the machine in real world trading
Post reply on HN