Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We gave 5 LLMs $100K to trade stocks for 8 months

aitradearena.com

Post reply on HN
Print view View original

81–90 of 319 posts

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#81

Post by jessetemp »

observationist wrote: post #50
olliepro wrote: post #19

Earlier quoted context omitted.

A more sound approach would have been to do a monte carlo simulation where you have 100 portfolios of each model and look at average performance.

Grok would likely have an advantage there, as well - it's got better coupling to X/Twitter, a better web search index, fewer safety guardrails in pretraining and system prompt modification that distort reality. It's easy to envision random market realities that would trigger ChatGPT or Claude into adjusting the output to be more politically correct. DeepSeek would be subject to the most pretraining distortion, but ha…

> fewer safety guardrails in pretraining and system prompt modification that distort reality.

Really? Isn't Grok's whole schtick that it's Elon's personal altipedia?

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#83

Post by alchemist1e9 »

cheeseblubber wrote: post #51
stusmall wrote: post #33

Earlier quoted context omitted.

Even if it is after the cut off date wouldn't the models be able to query external sources to get data that could positively impact them? If the returns were smaller I could reasonably believe it but beating the S&P500 returns by 4x+ strains credulity.

We used the LLMs API and provided custom tools like a stock ticker tool that only gave stock price information for that date of backtest for the model. We did this for news apis, technical indicator apis etc. It took quite a long time to make sure that there weren't any data leakage. The whole process took us about a month or two to build out.

I have a hunch Grok model cutoff is not accurate and somehow it has updated weights though they still call it the same Grok model as the params and size are unchanged but they are incrementally training it in the background. Of course I don’t know this but it’s what I would do in their situation since ongoing incremental training could he a neat trick to improve their ongoing results against competitors, even if marginal. I also wouldn’t trust the models to honestly disclose their decision process either.

That said. This is a fascinating area of research and I do think LLM driven fundamental investing and trading has a future.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#84

Post by alchemist1e9 »

endtime wrote: post #69
plufz wrote: post #9

Earlier quoted context omitted.

I know very little about how the environment where they run these models look, but surely they have access to different tools like vector embeddings with more current data on various topics?

If they could "see" the future and exploit that they'd probably have much higher returns.

56% over 8 months with the constraints provided are pretty good results for Grok.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#85

Post by alchemist1e9 »

disconcision wrote: post #71
disconcision wrote: post #53

Earlier quoted context omitted.

you can (via the api, or to a lesser degree through the setting in the web client) determine what tools if any a model can use

with the exception that it doesn't seem possible to fully disable this for grok 4

which is curiously the best model …

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#86

Post by Nevermark »

cheeseblubber wrote: post #70
Nevermark wrote: post #60

Just one run per model? That isn't backtesting. I mean technically it is, but "testing" implies producing meaningful measures. Also just one time interval? Something as trivial as "buy AI" could do well in one interval, and given models are going to be pumped about AI, ... 100 independent runs on each model over 10 very different market behavior time intervals would producing meaningful results. Like actually credibl…

Yes definitely we were using our own budget and out of our own pocket and these model runs were getting expensive. Claude costed us around 200-300 dollars a 8 month run for example. We want to scale it and get more statistically significant results but wanted to share something in the interim.

Got it. It is an interesting thing to explore.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#89

Post by nickthegreek »

jessetemp wrote: post #81
observationist wrote: post #50

Earlier quoted context omitted.

Grok would likely have an advantage there, as well - it's got better coupling to X/Twitter, a better web search index, fewer safety guardrails in pretraining and system prompt modification that distort reality. It's easy to envision random market realities that would trigger ChatGPT or Claude into adjusting the output to be more politically correct. DeepSeek would be subject to the most pretraining distortion, but ha…

> fewer safety guardrails in pretraining and system prompt modification that distort reality. Really? Isn't Grok's whole schtick that it's Elon's personal altipedia?

My understanding is that grok api is way different than the grok x bot. Which of course does Grok as a business any favors. Personally, I do not engage with either.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#90

Post by enlyth »

dash2 wrote: post #10

There's also this thing going on right now: https://nof1.ai/leaderboard Results are... underwhelming. All the AIs are focused on daytrading Mag7 stocks; almost all have lost money with gusto.

With the speed of how pricing information propagates, this seems way too dependent on how the agent is built, what information it has access to, and the feedback loop between the LLM and actions it can carry out
Post reply on HN