Model output is non-deterministic.
Did they make 10 calls per decision and then choose the majority? or did they just recreate the monkey picking stocks strategy?
We gave 5 LLMs $100K to trade stocks for 8 months
91–100 of 319 posts
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#92OP here. We realized there are a ton of limitations with backtest and paper money but still wanted to do this experiment and share the results. By no means is this statistically significant on whether or not these models can beat the market in the long term. But wanted to give everyone a way to see how these models think about and interact with the financial markets.
These are LLMs - next token guessers. They don't think at all and I suggest that you don't try to get rich quick with one!
LLMs are handy tools but no more. Even Qwen3-30B heavily quantised will do a passable effort of translating some Latin to English. It can whip up small games in a single prompt and much more and with care can deliver seriously decent results but so can my drill driver! That model only needs a £500 second hand GPU - that's impressive for me. Also GPT-OSS etc.
Yes, you can dive in with the bigger models that need serious hardware and they seem miraculous. A colleague had to recently "force" Claude to read some manuals until it realised it had made a mistake about something and frankly I think "it" was only saying it had made a mistake. I must ask said colleague to grab the reasoning and analyse it.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#93So.. I have been using an LLM to make 30 day buy and hold portfolios. And the results are "ok". (Like 8% vs 6% for the S&P 500 over the last 90 days) What you ask the model to do is super important. Just like writing or coding.. the default "behavior" is likely to be "average".. you need to very careful of what you are asking for. For me this is just a fun experiment and very interesting to see the market analysis it…
I look for issues with a recent double bottom and high insider buy activity. I've found this to be a highly reliable set of signals.
That is interesting.
I was trying to not be "very" prescriptive. My initial impression was, if you don't tell it to look at intrinsic value, the model will look at meme or very common stocks too much. Alternatively specifying an investing persona would probably also move it out of that default behavior profile. You have to kind of tell it about what it cares about. This isn't necessarily about trying to maximize a strategy, it was more about learning what kinds of things would it focus on, what kind of analysis.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#94How many trades? What's the z-score?
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#95For backtesting LLMs on polymarket I built. You can try with live data without sign up at: https://timba.fun
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#96Just one run per model? That isn't backtesting. I mean technically it is, but "testing" implies producing meaningful measures. Also just one time interval? Something as trivial as "buy AI" could do well in one interval, and given models are going to be pumped about AI, ... 100 independent runs on each model over 10 very different market behavior time intervals would producing meaningful results. Like actually credibl…
Yes, if these models available for $200/month a making 50% returns reliably, why isn’t Citadel having layoffs?
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#97OP here. We realized there are a ton of limitations with backtest and paper money but still wanted to do this experiment and share the results. By no means is this statistically significant on whether or not these models can beat the market in the long term. But wanted to give everyone a way to see how these models think about and interact with the financial markets.
You should redo this with human controls. By a weird coincidence, I have sufficient free time.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#98Earlier quoted context omitted.
Yeah I mean if you generally believe the tech sector is going to do well because it has been doing well you will beat the overall market. The problem is that you don’t know if and when there might be a correction. But since there is this one segment of the overall market that has this steady upwards trend and it hasn’t had a large crash, then yeah any pattern seeking system will identify “hey this line keeps going up…
> It would almost be more interesting to specifically train the model on half the available market data, then test it on another half. Yes, ideally you’d have a model trained only on data up to some date, say January 1, 2010, and then start running the agents in a simulation where you give them each day’s new data (news, stock prices, etc.) one day at a time.
I mean ultimately this is an exercise in frustration because if you do that you will have trained your model on market patterns that might not be in place anymore. For example after the 2008 recession regulations changed. So do market dynamics actually work the same in 2025 as in 2005? I honestly don’t know but intuitively I would say that it is possible that they do not.
I think a potentially better way would be to segment the market up to today but take half or 10% of all the stocks and make only those available to the LLM. Then run the test on the rest. This accounts for rules and external forces changing how markets operate over time. And you can do this over and over picking a different 10% market slice for training data each time.
But then your problem is that if you exclude let’s say Intel from your training data and AMD from your testing data then there ups and downs don’t really make sense since they are direct competitors. If you separate by market segment then does training the model on software tech companies might not actually tell you accurately how it would do for commodities or currency training. Or maybe I am wrong and trading is trading no matter what you are trading.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#99Earlier quoted context omitted.
> fewer safety guardrails in pretraining and system prompt modification that distort reality. Really? Isn't Grok's whole schtick that it's Elon's personal altipedia?
My understanding is that grok api is way different than the grok x bot. Which of course does Grok as a business any favors. Personally, I do not engage with either.
you gotta be quite a crazy person to use grok :)
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#100we should:
1. train with a cutoff date at ~2006
2. simulate information flow (financial data, news, earnings, ...) day by day
3. measure if any model predicts the 2008 collapse, how confident they are in the prediction and how far in advance