Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploring the limits of large language models as quant traders

nof1.ai

Post reply on HN
Print view View original

81–90 of 103 posts

Re: Exploring the limits of large language models as quant traders

#81

Post by pants2 »

kqr wrote: post #2

Super interesting! You can click the "live" link in the header to see how they performed over time. The (geometric) average result at the end seems to be that the LLMs are down 35 % from their initial capital – and they got there in just 96 model-days. That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital. Although I lack the maths to determine it numerically (…

Well, if you can get a model to consistently lose money like that, then you just trade the opposite of what it says and you're guaranteed money!

Re: Exploring the limits of large language models as quant traders

#82

Post by pants2 »

Libidinalecon wrote: post #57
lordnacho wrote: post #33

I was chatting to a friend in the space. This guy is both experienced in trading and LLMs, and has gone all-in on using LLMs to get his day-to-day coding done. Now he's working on the model to end all models, which is a fairly ambitious way to put it, but it throws off some interesting conversations. You need domain knowledge to get this to work. Things like "we fed the model the market data" are actually non-obvious…

You can vibe code in this space as an individual because practically everything you are going to write is already in the training data. The big Quant hedge funds have been using machine learning for decades. I took the coursera RL in finance class years ago. The idea you are going to beat Two Sigma at their own game with tokens is just an absurdity. Personally, I think any individual on their own that claims they are…

> The idea you are going to beat Two Sigma at their own game with tokens is just an absurdity.

Individual quant traders aren't competing with Two Sigma. If you're an individual quant trader and you find a signal with $500k/yr capacity, that's awesome. If you're Two Sigma you won't give a single cahoot if it's not a $50M/yr signal. Two completely different ball games. I doubt Two Sigma is even trading on Hyperliquid either.

Re: Exploring the limits of large language models as quant traders

#83

Post by EMM_386 »

IAmGraydon wrote: post #43

Crazy how people continue to treat LLMs like they’re anything more than a record of past human knowledge and are then surprised when they can’t predict the future.

Humans don't trade on future knowledge either.

Well, most of them - that can be illegal.

Re: Exploring the limits of large language models as quant traders

#84

Post by kqr »

pants2 wrote: post #81
kqr wrote: post #2

Super interesting! You can click the "live" link in the header to see how they performed over time. The (geometric) average result at the end seems to be that the LLMs are down 35 % from their initial capital – and they got there in just 96 model-days. That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital. Although I lack the maths to determine it numerically (…

Well, if you can get a model to consistently lose money like that, then you just trade the opposite of what it says and you're guaranteed money!

Thanks to the magic of compounding, inverting overbetting also leads to overbetting. Especially once costs are accounted for.

Re: Exploring the limits of large language models as quant traders

#85

Post by infecto »

This might be the dumbest thing I have ever seen but I am happy to be corrected and told why it’s not.

I use LLMs a lot and I work in finance and I don’t see how a LLM benefits in this space.

Also it looks like none of their data uses any kind of benchmarking. It’s purely a which model did better which I don’t think tells you much.

Re: Exploring the limits of large language models as quant traders

#86

Post by ribosometronome »

aswegs8 wrote: post #13

Given that LLMs can't even finish Pokemon Red, how would you expect they are able to trade futures?

About that...

https://www.reddit.com/r/ClaudePlaysPokemon/comments/1otd4kl...

seems like the big issue is just spending time with the tooling to interact with Pokemon and just that calling an LLM for each button is time consuming.

Re: Exploring the limits of large language models as quant traders

#87

Post by infecto »

baq wrote: post #8
Havoc wrote: post #7

Are language models really the best choice for this? Seems to me that the outcome would be near random because they are so poorly suited. Which might manifest as > We also found that the models were highly sensitive to seemingly trivial prompt changes

they're tools. treat them as tools. since they're so general, you need to explore if and how you can use them in your domain. guessing 'they're poorly suited' is just that, guessing. in particular: > We also found that the models were highly sensitive to seemingly trivial prompt changes this is as much as obvious for anyone who seriously looked at deploying these, that's why there are some very successful startups in…

I agree in sentiment but if you spent any amount of time in finance, even outside of equity markets, you would have a pretty quick mental model that LLMs are a weird fit for this space.

Re: Exploring the limits of large language models as quant traders

#88

Post by pinkmuffinere »

> Ordering bias. Early prompts listed market data newest→oldest. Even with explicit notes, several models still read it as oldest → newest, inferring the wrong state. Switching to oldest → newest fixed the immediate error and suggests a formatting prior in current LLMs.

This kind of error just feels comical to me, and really makes it hard for me to believe that AGI is anywhere near. LLM's struggle to understand the order of datasets, when explicitly told. This is like showing a coin trick to a child, except perhaps even simpler.

Re: Exploring the limits of large language models as quant traders

#90

Post by ramoz »

thisisit wrote: post #62

LLMs are very good at NLP/classification tasks and weak at calculations and numbers. So, I doubt feeding it numerical data is a good idea. And if you feeding or harnessing as the blog post puts it in a way that where it reasons things like: > RSI 7-period: 62.5 (neutral-bullish) Then it is no better than normal automated trading where the program logic is something along the lines if RSI > 80 then exit. And looking a…

You wouldnt feed it numerical data, but you would allow it to make certain calculations (via tools of a harness) as it relates to your portfolio.
Post reply on HN