Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We gave 5 LLMs $100K to trade stocks for 8 months

aitradearena.com

Post reply on HN
Print view View original

51–60 of 319 posts

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#51

Post by cheeseblubber »

stusmall wrote: post #33
PTRFRLL wrote: post #4

Earlier quoted context omitted.

> We were cautious to only run after each model’s training cutoff dates for the LLM models. That way we could be sure models couldn’t have memorized market outcomes.

Even if it is after the cut off date wouldn't the models be able to query external sources to get data that could positively impact them? If the returns were smaller I could reasonably believe it but beating the S&P500 returns by 4x+ strains credulity.

We used the LLMs API and provided custom tools like a stock ticker tool that only gave stock price information for that date of backtest for the model. We did this for news apis, technical indicator apis etc. It took quite a long time to make sure that there weren't any data leakage. The whole process took us about a month or two to build out.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#52

Post by jondwillis »

1a527dd5 wrote: post #46

Time. That has been the best way to get returns. I setup a 212 account when I was looking to buy our first house. I bought in small tiny chunks of industry where I was comfortable and knowledgeable in. Over the years I worked up a nice portfolio. Anyway, long story short. I forgot about the account, we moved in, got a dog, had children. And then I logged in for the first time in ages, and to my shock. My returns were…

…did you beat the market? 110% is pretty much what the nasdaq has done over the last 5 years

Also N=1

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#53

Post by disconcision »

plufz wrote: post #9
PTRFRLL wrote: post #4

Earlier quoted context omitted.

> We were cautious to only run after each model’s training cutoff dates for the LLM models. That way we could be sure models couldn’t have memorized market outcomes.

I know very little about how the environment where they run these models look, but surely they have access to different tools like vector embeddings with more current data on various topics?

you can (via the api, or to a lesser degree through the setting in the web client) determine what tools if any a model can use

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#54

Post by cheeseblubber »

mlmonkey wrote: post #40

> We were cautious to only run after each model’s training cutoff dates for the LLM models Grok is constantly training and/or it has access to websearch internally. You cannot backtest LLMs. You can only "live" test them going forward.

Via api you can turn off websearch internally. We provided all the models with their own custom tools that only provided data up to the date of the backtest.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#55

Post by DennisP »

JoeAltmaier wrote: post #45
etchalon wrote: post #18

Earlier quoted context omitted.

I don't feel like they measured anything. They just confirmed that tech stocks in the US did pretty well.

They measured the investment facility of all those LLMs. That's pretty much what the title says. And they had dramatically different outcomes. So that tells me something.

I mean, what it kinda tells me is that people talk about tech stocks the most, so that's what was most prevalent in the training data, so that's what most of the LLMs said to invest in. That's the kind of strategy that works until it really doesn't.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#58

Post by joegibbs »

cheeseblubber wrote: post #39

OP here. We realized there are a ton of limitations with backtest and paper money but still wanted to do this experiment and share the results. By no means is this statistically significant on whether or not these models can beat the market in the long term. But wanted to give everyone a way to see how these models think about and interact with the financial markets.

I think it would be interesting to see how it goes in a scenario where the market declines or where tech companies underperform the rest of the market. In recent history they've outperformed the market and that might bias the choices that the LLMs make - would they continue with these positive biases if they were performing badly?

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#60

Post by Nevermark »

Just one run per model? That isn't backtesting. I mean technically it is, but "testing" implies producing meaningful measures.

Also just one time interval? Something as trivial as "buy AI" could do well in one interval, and given models are going to be pumped about AI, ...

100 independent runs on each model over 10 very different market behavior time intervals would producing meaningful results. Like actually credible, meaningful means and standard deviations.

This experiment, as is, is a very expensive unbalanced uncharacterizable random number generator.

Post reply on HN