Live data from Hacker News

We gave 5 LLMs $100K to trade stocks for 8 months

aitradearena.com

241–250 of 319 posts

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#241
post #236

Predicting the stock market will likely never happen because it’s recursive. We can predict the next 10 days of weather, but the weather doesn’t change because it read your forecast. As long as markets continue to react to their own reactions, they will remain unpredictable. If the strategy is long, there might be alpha to be found. But day trading? No way.

If stocks are more of a closed system that are weakly affected by external factors in the short term, now I finally understand why they hire so many physicists for financial modeling!

There is of course the fact that physicists tend to be the best applied mathematicians, even if they don’t end up using any of their physics knowledge. And they generally had the reputation of “the smartest” people for the last century.

Anyway, such systems are complex and chaotic yes, but there are many ways of predicting aspects of them, like with fluid simulation to give a basic example. And I don’t get your point about weather, it is also recursive in the same way and reacting to its own reactions. Sure it is not reacting to predictions of itself, but that’s just a special kind of reaction, and patterns in others predictions can definitely be predicted accurately, perhaps not individually but in the aggregate.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#242

Earlier quoted context omitted.

>but for various reasons can end up flopping when actually executed in the real market. 1. Your order can legally be “front run” by the lead or designated market maker who receives priority trade matching, bypassing the normal FIFO queue. Not all exchanges do this. 2. Market impact. Other participants will cancel their order, or increase their order size, based on your new order. And yes, the algos do care about your…

If you actually were in the industry, you would know that most retail traders don't fail, because they lose a tick here or there on execution, they fail, because their strategies have no edge in the first place.

> If you actually were in the industry, you would know that most retail traders don't fail, because they lose a tick here or there on execution

Where did I say “retail trader”?

Because “institutional” low-latency market makers trade 1 lot all the time.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#243

> Grok ended up performing the best while DeepSeek came close to second. Almost all the models had a tech-heavy portfolio which led them to do well. Gemini ended up in last place since it was the only one that had a large portfolio of non-tech stocks. I'm not an investor or researcher, but this triggers my spidey sense... it seems to imply they aren't measuring what they think they are.

probably hitching onto sycophancy for the parent company and getting lucky as a result... that Grok September rally aligns somewhat with TSLA for instance

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#244
post #236

Predicting the stock market will likely never happen because it’s recursive. We can predict the next 10 days of weather, but the weather doesn’t change because it read your forecast. As long as markets continue to react to their own reactions, they will remain unpredictable. If the strategy is long, there might be alpha to be found. But day trading? No way.

"We can predict the next 10 days of weather, but the weather doesn’t change because it read your forecast."

Less true than it used to be, with cloud seeding being an off-the-shelf technology now. Still largely true, but not entirely true anymore.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#245

This is pretty cool. We're also running a live experiment on both stocks and options. One difference with our experiment is a lot more tools being available to the models (anything you can think of, sec filings, fundamentals, live pricing, options data). We think backtests are meaningless given LLMs have mostly memorized every single thing that happened so it's not a good test. So we're running a forward test. Not en…

How is Qwen so much worse than the rest (for the period accounted)?

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#246

OP here. We realized there are a ton of limitations with backtest and paper money but still wanted to do this experiment and share the results. By no means is this statistically significant on whether or not these models can beat the market in the long term. But wanted to give everyone a way to see how these models think about and interact with the financial markets.

You're not really giving them any money and it's not actually trading.

There's no market impact to any trading decision they make.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#247

Earlier quoted context omitted.

> But wanted to give everyone a way to see how these models think… Think? What exactly did “it” think about?

You can click in to the chart and see the conversation as well as for each trade what was the reasoning it gave for it

A model can't tell you why it made the decision.

What it can do is inspect the decision it made and make up a reason a human might have said when making the decision.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#248
post #108

I wouldn’t trust any backtracking test with these models. Try doing a real-time test over 8 months and see what happens then. I’d also be suspicious of anything that doesn’t take actual costs into account.

We're running some live experiments these days, for both stocks and options. https://rallies.ai/arena

With actual money? Or still fake money?

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#249
post #236

Predicting the stock market will likely never happen because it’s recursive. We can predict the next 10 days of weather, but the weather doesn’t change because it read your forecast. As long as markets continue to react to their own reactions, they will remain unpredictable. If the strategy is long, there might be alpha to be found. But day trading? No way.

If stocks are more of a closed system that are weakly affected by external factors in the short term, now I finally understand why they hire so many physicists for financial modeling! There is of course the fact that physicists tend to be the best applied mathematicians, even if they don’t end up using any of their physics knowledge. And they generally had the reputation of “the smartest” people for the last century.…

> there are many ways of predicting aspects of them

Yes, and it's priced in

> but that’s just a special kind of reaction

That's just arguing semantics. My point was that weather doesn't react to human predictions, explicitly

Post reply on HN