I appreciate that you’ve made the trade histories downloadable and will be taking a look to see what I can learn.
I’ve glanced over some of it and really wonder why they seemed to focus on a small group of stocks.
We gave 5 LLMs $100K to trade stocks for 8 months
171–180 of 319 posts
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#172Why is bullshit detector ringing as hell right now??? This sounds like another billion-dollar-Markov-chain-IP that claimed to change the world, opening with a paper with flying colors.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#173I used to work for a brokerage API geared at algorithmic traders and in my experience anecdotal experience many strategies seem to work well when back-tested on paper but for various reasons can end up flopping when actually executed in the real market. Even testing a strategy in real time paper trading can end up differently than testing on the actual market where other parties are also viewing your trades and makin…
>but for various reasons can end up flopping when actually executed in the real market. 1. Your order can legally be “front run” by the lead or designated market maker who receives priority trade matching, bypassing the normal FIFO queue. Not all exchanges do this. 2. Market impact. Other participants will cancel their order, or increase their order size, based on your new order. And yes, the algos do care about your…
Dear HFT Quant,
> And yes, the algos do care about your little 1 lot order.
I'm just your usual "corrupted nerd" geek with some mathematics and computer security background interests - 2 questions if I may 1. what's like the most interesting paper you have read recently or unrelated thing you are interested in at the moment? 2. " And yes, the algos do care about your little 1 lot order." How would one see this effect you mentioned - like it seems wildly anomalous, how would go about finding this effect assuming maximum mental venturesomeness, a tiny $100 and too much time?
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#174lolol Gemini
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#175Earlier quoted context omitted.
Yeah I mean if you generally believe the tech sector is going to do well because it has been doing well you will beat the overall market. The problem is that you don’t know if and when there might be a correction. But since there is this one segment of the overall market that has this steady upwards trend and it hasn’t had a large crash, then yeah any pattern seeking system will identify “hey this line keeps going up…
> It would almost be more interesting to specifically train the model on half the available market data, then test it on another half. Yes, ideally you’d have a model trained only on data up to some date, say January 1, 2010, and then start running the agents in a simulation where you give them each day’s new data (news, stock prices, etc.) one day at a time.
I suspect trading firms have already done this to the maximum extent that it's profitable to do so. I think if you were to integrate LLMs into a trading algorithm, you would need to incorporate more than just signals from the market itself. For example, I hazard a guess you could outperform a model that operates purely on market data with a model that also includes a vector embedding of a selection of key social and news media accounts or other information sources that have historically been difficult to encode until LLMs.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#176This is pretty cool.
We're also running a live experiment on both stocks and options. One difference with our experiment is a lot more tools being available to the models (anything you can think of, sec filings, fundamentals, live pricing, options data).
We think backtests are meaningless given LLMs have mostly memorized every single thing that happened so it's not a good test. So we're running a forward test. Not enough data for now but pretty interesting initial results
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#177There's also this thing going on right now: https://nof1.ai/leaderboard Results are... underwhelming. All the AIs are focused on daytrading Mag7 stocks; almost all have lost money with gusto.
I think the big limitation of nof1 is that they're not using a lot of data that an actual investor would use when researching companies.
We're trying to fix some of those limitations and run a similar live competition at https://rallies.ai/arena
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#178I wouldn’t trust any backtracking test with these models. Try doing a real-time test over 8 months and see what happens then. I’d also be suspicious of anything that doesn’t take actual costs into account.
We're running some live experiments these days, for both stocks and options.
https://rallies.ai/arena
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#179Earlier quoted context omitted.
Grok would likely have an advantage there, as well - it's got better coupling to X/Twitter, a better web search index, fewer safety guardrails in pretraining and system prompt modification that distort reality. It's easy to envision random market realities that would trigger ChatGPT or Claude into adjusting the output to be more politically correct. DeepSeek would be subject to the most pretraining distortion, but ha…
OTOH it has the richest man in the world actively meddling in its results when they don't support his politics.
Anyone who hasn't used Grok might be surprised to learn that it isn't shy about disagreeing with Elon on plenty of topics, political or otherwise. Any insinuation to the contrary seems to be pure marketing spin on his part.
Grok is often absurdly competent compared to other SOTA models, definitely not a tool I'd write off over its supposed political leanings. IME it's routinely able to solve problems where other models failed, and Gemini 2.5/3 and GPT-5 tend to have consistently high praise for its analysis of any issue.
That's as far as the base model/chatbot is concerned, at least. I'm less familiar with the X bot's work.
Re: We gave 5 LLMs $100K to trade stocks for 8 months
#180Earlier quoted context omitted.
> It would almost be more interesting to specifically train the model on half the available market data, then test it on another half. Yes, ideally you’d have a model trained only on data up to some date, say January 1, 2010, and then start running the agents in a simulation where you give them each day’s new data (news, stock prices, etc.) one day at a time.
I mean ultimately this is an exercise in frustration because if you do that you will have trained your model on market patterns that might not be in place anymore. For example after the 2008 recession regulations changed. So do market dynamics actually work the same in 2025 as in 2005? I honestly don’t know but intuitively I would say that it is possible that they do not. I think a potentially better way would be to…
> I think a potentially better way would be to segment the market up to today but take half or 10% of all the stocks and make only those available to the LLM.
Those stocks are going to be coupled. Let's take an easy example. Suppose you include Nvidia in the training data and hold out AMD for test. Is there information leakage? Yes. The problem is that each company isn't independent. You have information leakage in both the setting where companies grow together as well as zero sum games (since x + y = 0, if you know x then you know y). But in this example AMD tends with Nvidia. Maybe not as much, but they go in the same direction. They're coupled
Not to mention that in the specific setting the LLMs were given news and other information.