Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We gave 5 LLMs $100K to trade stocks for 8 months

aitradearena.com

Post reply on HN
Print view View original

221–230 of 319 posts

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#221

Post by throwawayffffas »

tclancy wrote: post #118
bcrosby95 wrote: post #12

> Grok ended up performing the best while DeepSeek came close to second. Almost all the models had a tech-heavy portfolio which led them to do well. Gemini ended up in last place since it was the only one that had a large portfolio of non-tech stocks. I'm not an investor or researcher, but this triggers my spidey sense... it seems to imply they aren't measuring what they think they are.

I mean, run the experiment during a different trend in the market and the results would probably be wildly different. This feels like chartists [1] but lazier. [1] https://www.investopedia.com/terms/c/chartist.asp

To be fair to chartists, they try to identify if they are in a bear market or one is coming and get out early.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#223

Post by throwawayffffas »

bmitc wrote: post #143
culi wrote: post #128

Earlier quoted context omitted.

this study should be replicated during a bear market

Buy and hold performs well over long time scales by simply not adjusting based upon sentiment.

Operating word is long, historically if you entered the market just before a downturn, it could take years up to a couple of decades to make up. Depending on which downturn we are looking at.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#224

Post by skeeter2020 »

Eddy_Viscosity2 wrote: post #209
IgorPartola wrote: post #34

Earlier quoted context omitted.

Yeah I mean if you generally believe the tech sector is going to do well because it has been doing well you will beat the overall market. The problem is that you don’t know if and when there might be a correction. But since there is this one segment of the overall market that has this steady upwards trend and it hasn’t had a large crash, then yeah any pattern seeking system will identify “hey this line keeps going up…

> a hedge fund can beat the market for 2-4 years but at 10 years and up their chances of beating the market go to very close In that case the winning strategy would be to switch hedge funds every 3 years.

Except you don't know which fund is going to "go on a hot streak" or when the magic will end. The original statement only holds when looking at historical data; it's not predictive.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#225

Post by devilsbabe »

throwawayffffas wrote: post #219

> We also built a way to simulate what an agent would have seen at any point in the past. Each model gets access to market data, news APIs, company financials—but all time filtered: agents see only what would have been available on that specific day during the test period. That's not going to work, these agents especially the larger ones, will have news about the companies embedded in their weights.

Funny how if you kept reading before commenting, they addressed that point specifically

> We were cautious to only run after each model’s training cutoff dates for the LLM models. That way we could be sure models couldn’t have memorized market outcomes.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#226

Post by skeeter2020 »

buu700 wrote: post #179
KPGv2 wrote: post #162

Earlier quoted context omitted.

OTOH it has the richest man in the world actively meddling in its results when they don't support his politics.

Anyone who hasn't used Grok might be surprised to learn that it isn't shy about disagreeing with Elon on plenty of topics, political or otherwise. Any insinuation to the contrary seems to be pure marketing spin on his part. Grok is often absurdly competent compared to other SOTA models, definitely not a tool I'd write off over its supposed political leanings. IME it's routinely able to solve problems where other mode…

it's so wildly inconsistent you can't build on top of it with reliability. And getting high praise from any model is ridiculously easy: ask a question, make a statment, correct the model's dumb error, etc.

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#227

Post by skeeter2020 »

KPGv2 wrote: post #163
bdangubic wrote: post #99

Earlier quoted context omitted.

you gotta be quite a crazy person to use grok :)

I sat in my kid's extracurricular a couple months ago and had an FBI agent tell me that Grok was the most trustworthy based on "studies," so that's what she had for her office.

Did she get that info from Grok?

Re: We gave 5 LLMs $100K to trade stocks for 8 months

#228

Post by skeeter2020 »

JoeAltmaier wrote: post #45
etchalon wrote: post #18

Earlier quoted context omitted.

I don't feel like they measured anything. They just confirmed that tech stocks in the US did pretty well.

They measured the investment facility of all those LLMs. That's pretty much what the title says. And they had dramatically different outcomes. So that tells me something.

They "proved" that US tech stocks did better than portfolios with less US tech stocks over a recent, very short time range. 1. You didn't know that? 2. Whata re you going to do with this "new information"?
Post reply on HN