This experiment was also performed with a fish [1] though it was only given $50,000. Spoiler, the fish did great vs wall street bets.
[1] - https://www.youtube.com/watch?v=USKD3vPD6ZA [video][15 mins]
We gave 5 LLMs $100K to trade stocks for 8 months
61–70 of 319 posts
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#62I used to work for a brokerage API geared at algorithmic traders and in my experience anecdotal experience many strategies seem to work well when back-tested on paper but for various reasons can end up flopping when actually executed in the real market. Even testing a strategy in real time paper trading can end up differently than testing on the actual market where other parties are also viewing your trades and making their own responses. The post did list some potential disadvantages of backtesting, so they clearly aren't totally in the dark on it.
Deepseek did not sell anything, but did well with holding a lot of tech stocks. I think that can be a bit of a risky strategy with everything in one sector, but it has been a successful one recently so not surprising that it performed well. Seems like they only get to "trade" once per day, near the market close, so it's not really a real time ingesting of data and making decisions based on that.
What would really be interesting is if one of the LLMs switched their strategy to another sector at an appropriate time. Very hard to do but very impressive if done correctly. I didn't see that anywhere but I also didn't look deeply at every single trade.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#63Could they give some random people (i volunteer) 100k for 8 months? ...as a control
I know this is a joke comment, but there are plenty of websites that simulate the stock market and where you can use paper money to trade. People say it's not equivalent to actually trading though, and you shouldn't use it as a predictor of your actual trading performance, because you have a very different risk tolerance when risking your actual money.
Yeah, if you give me $100K I'm almost certainly going to make very different decisions than either a supposedly optimizing computer or myself at different ages.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#64> Grok ended up performing the best while DeepSeek came close to second. Almost all the models had a tech-heavy portfolio which led them to do well. Gemini ended up in last place since it was the only one that had a large portfolio of non-tech stocks. I'm not an investor or researcher, but this triggers my spidey sense... it seems to imply they aren't measuring what they think they are.
They're not measuring performance in the context of when things happen and in the time that they are. It think its only showing recent performance and popularity. To actually evaluate how these do you need to be able to correct the model and retrain it per different time periods and then measure how it would do. Then you'll get better information from the backtesting.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#65Time. That has been the best way to get returns. I setup a 212 account when I was looking to buy our first house. I bought in small tiny chunks of industry where I was comfortable and knowledgeable in. Over the years I worked up a nice portfolio. Anyway, long story short. I forgot about the account, we moved in, got a dog, had children. And then I logged in for the first time in ages, and to my shock. My returns were…
time in the market beats timing the market -> Kenneth Fisher ... i learned it the hard way ;)
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#66OP here. We realized there are a ton of limitations with backtest and paper money but still wanted to do this experiment and share the results. By no means is this statistically significant on whether or not these models can beat the market in the long term. But wanted to give everyone a way to see how these models think about and interact with the financial markets.
> Grok ended up performing the best while DeepSeek came close to second.
I think you mean "DeepSeek came in a close second".
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#67> Grok ended up performing the best while DeepSeek came close to second. Almost all the models had a tech-heavy portfolio which led them to do well. Gemini ended up in last place since it was the only one that had a large portfolio of non-tech stocks. I'm not an investor or researcher, but this triggers my spidey sense... it seems to imply they aren't measuring what they think they are.
Yeah I mean if you generally believe the tech sector is going to do well because it has been doing well you will beat the overall market. The problem is that you don’t know if and when there might be a correction. But since there is this one segment of the overall market that has this steady upwards trend and it hasn’t had a large crash, then yeah any pattern seeking system will identify “hey this line keeps going up…
> It would almost be more interesting to specifically train the model on half the available market data, then test it on another half.
Yes, ideally you’d have a model trained only on data up to some date, say January 1, 2010, and then start running the agents in a simulation where you give them each day’s new data (news, stock prices, etc.) one day at a time.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#68“Everyone (including LLMs) is a genius in a bull market.”
Apparently everyone (but Gemini).
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#69Earlier quoted context omitted.
> We were cautious to only run after each model’s training cutoff dates for the LLM models. That way we could be sure models couldn’t have memorized market outcomes.
I know very little about how the environment where they run these models look, but surely they have access to different tools like vector embeddings with more current data on various topics?
If they could "see" the future and exploit that they'd probably have much higher returns.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#70Just one run per model? That isn't backtesting. I mean technically it is, but "testing" implies producing meaningful measures. Also just one time interval? Something as trivial as "buy AI" could do well in one interval, and given models are going to be pumped about AI, ... 100 independent runs on each model over 10 very different market behavior time intervals would producing meaningful results. Like actually credibl…
Yes definitely we were using our own budget and out of our own pocket and these model runs were getting expensive. Claude costed us around 200-300 dollars a 8 month run for example. We want to scale it and get more statistically significant results but wanted to share something in the interim.