Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploring the limits of large language models as quant traders

nof1.ai

Post reply on HN
Print view View original

91–100 of 103 posts

Re: Exploring the limits of large language models as quant traders

#91

Post by p1necone »

pinkmuffinere wrote: post #88

> Ordering bias. Early prompts listed market data newest→oldest. Even with explicit notes, several models still read it as oldest → newest, inferring the wrong state. Switching to oldest → newest fixed the immediate error and suggests a formatting prior in current LLMs. This kind of error just feels comical to me, and really makes it hard for me to believe that AGI is anywhere near. LLM's struggle to understand the o…

I've noticed similar issues to this with rendering related code. Most models had a strong preference for z-up over z-down (I think, it might have been the other way around), and correcting them only fixed it for maybe the next response then the model would go back to using the wrong coordinates and getting confused by the outcome.

No amount of added context or instructions seems to fix these kind of issues in a way that doesn't still feel pretty hobbled. The only way to get the full power out of the model is to conform your problem to the expectations that seem to be baked in - i.e. just change your rendering coordinate system to be z-up.

Re: Exploring the limits of large language models as quant traders

#92

Post by achierius »

ponector wrote: post #80
kqr wrote: post #2

Super interesting! You can click the "live" link in the header to see how they performed over time. The (geometric) average result at the end seems to be that the LLMs are down 35 % from their initial capital – and they got there in just 96 model-days. That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital. Although I lack the maths to determine it numerically (…

>> That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital LLM indeed can replace average human being.

The average human wipes out their retirements savings in one year?

Re: Exploring the limits of large language models as quant traders

#93

Post by devin »

When I saw this I rolled my eyes. It is well-understood that purpose-built models perform better than general models on tasks like this, and yet it would seem one of the main purposes of running this experiment according to the website is to figure out if general models are enough.

In addition, I cannot imagine how the selection of securities was chosen. Is XRP seriously part of the proposed asset mix here?

It's hard not to look at this and view it as a marketing stunt. Nothing about the results are surprising and the setup does not seem to make any sense to me to begin with.

Re: Exploring the limits of large language models as quant traders

#94

Post by cauefcr »

achierius wrote: post #92
ponector wrote: post #80

Earlier quoted context omitted.

>> That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital LLM indeed can replace average human being.

The average human wipes out their retirements savings in one year?

When they try to day trade, yes they do.

Re: Exploring the limits of large language models as quant traders

#95

Post by xwolfi »

ta12653421 wrote: post #65
callamdelaney wrote: post #12

The limits of LLM's for systematic trading were and are extremely obvious to anybody with a basic understanding of either field. You may as well be flipping a coin.

In general, I agree - but there is one exception, I think: However you put AI into an stat arb context, I think it may help for trading on a daily base like "tell me where i should enter this morning and exit this evening". (not daytrading throughout the whole day) But, I havent tested it so far since I do not believe it either :D

Why would it know anything better than a bunch of 12 yo given the same question ? LLM don't know things very well, they don't cross concepts in their mind. Give you an example, made $1500 yday trading nvidia:

I followed the curve for the last month, scalping a few times - I get a feel like panic point is ~180$, hype point ~195$, it's like that most swings. There were earnings yday, people are afraid that the company is over its head already and prefer to de-risk, which I do too sometimes on other stuff. It is true that nvidia is overpriced ofc, but I feel we have maybe a few good runs and that's where the risk, therefore the potential reward, is. I enter around 184, and a bit more around 182. I go to sleep (Im in China), and when I wake up I sell at 194. I got lucky, and I would not do it again before I understand why would nvidia be swinging again.

Is an LLM gonna be any better ? My brain did a classic Bayes analysis, used the recent past as strong signal to my prediction of the future (a completely absurd bias ofc, but all traders are absurd humans), I played a company that wasnt gonna burn me too much, since Im still happy to own shares of nvidia whatever the price, and the money put there was losable entirely without too much pain.

Do I need AI ? Meh. For your next play, do you trust me or chatgpt more ? I can explain my decisions very coherently, with good caveats on my limits and biases, and warnings about what risk to afford when. I experienced losses and gains, and I know the effect and causes of both, and how to deal with them both. I prefer me, to it.

Re: Exploring the limits of large language models as quant traders

#96

Post by potato3732842 »

Libidinalecon wrote: post #57
lordnacho wrote: post #33

I was chatting to a friend in the space. This guy is both experienced in trading and LLMs, and has gone all-in on using LLMs to get his day-to-day coding done. Now he's working on the model to end all models, which is a fairly ambitious way to put it, but it throws off some interesting conversations. You need domain knowledge to get this to work. Things like "we fed the model the market data" are actually non-obvious…

You can vibe code in this space as an individual because practically everything you are going to write is already in the training data. The big Quant hedge funds have been using machine learning for decades. I took the coursera RL in finance class years ago. The idea you are going to beat Two Sigma at their own game with tokens is just an absurdity. Personally, I think any individual on their own that claims they are…

>The idea you are going to beat Two Sigma at their own game with tokens is just an absurdity.

The idea isn't to beat them. It's to pick up the scraps. Same as every small trading operation.

I've seen the books of a guy who makes money hand over fist trading options. He'll be the first one to tell you what he does won't scale.

Re: Exploring the limits of large language models as quant traders

#97

Post by beefnugs »

lordnacho wrote: post #33

I was chatting to a friend in the space. This guy is both experienced in trading and LLMs, and has gone all-in on using LLMs to get his day-to-day coding done. Now he's working on the model to end all models, which is a fairly ambitious way to put it, but it throws off some interesting conversations. You need domain knowledge to get this to work. Things like "we fed the model the market data" are actually non-obvious…

These kinds of tests to me are not complete until they resolve the concept to full solution:

-Start just as they have here

-Keep improving the prompts in a huge variety of ways to see what improvements can be made

-start getting more and more code generated to complete more and more percentage of the work instead of textual prompting

-start fixing the worst parts with real human knowledge code/tools

-finally show fully working solution that does well, with full analysis of what kind of human intervention was necessary, and even explore what kind of prompting could lead to these human intuition-ed tooling going to whatever incredible lengths necessary to hand-hold the models in the right direction

otherwise... i don't get the points of stopping and saying "doesn't do great"

Re: Exploring the limits of large language models as quant traders

#98

Post by ta12653421 »

xwolfi wrote: post #95
ta12653421 wrote: post #65

Earlier quoted context omitted.

In general, I agree - but there is one exception, I think: However you put AI into an stat arb context, I think it may help for trading on a daily base like "tell me where i should enter this morning and exit this evening". (not daytrading throughout the whole day) But, I havent tested it so far since I do not believe it either :D

Why would it know anything better than a bunch of 12 yo given the same question ? LLM don't know things very well, they don't cross concepts in their mind. Give you an example, made $1500 yday trading nvidia: I followed the curve for the last month, scalping a few times - I get a feel like panic point is ~180$, hype point ~195$, it's like that most swings. There were earnings yday, people are afraid that the company…

you apply it to find cross correlation ideas about larger numbers of assets. Try doing your stuff on a daily base with more than 500 assets :-)

Re: Exploring the limits of large language models as quant traders

#99

Post by lambdaone »

falcor84 wrote: post #77
callamdelaney wrote: post #72

Earlier quoted context omitted.

The only applications of generative AI I can envisage for trading, systematically or otherwise are the following: - data extraction: It's possible to get pretty good levels of accuracy on unstructured data, eg financial reports with relatively little effort compared to before decent llm's - sentiment analysis: Why bother with complicated sentiment analysis when you can just feed an article into an LLM for scoring? -…

> If an llm is fed on losing / scammy rubbish, how could it possibly produce a return? Rather than just relying on pretraining, you'd use RL on the trade outcomes.

RL would reasonably be expected to work if the market had some sort of discoverable static behavior.

The reason why RL by backtesting cannot work is that the real market is continuously changing, as all the agents within it, both human and automated, are constantly updating their opinions and strategies.

Re: Exploring the limits of large language models as quant traders

#100

Post by xwolfi »

ta12653421 wrote: post #98
xwolfi wrote: post #95

Earlier quoted context omitted.

Why would it know anything better than a bunch of 12 yo given the same question ? LLM don't know things very well, they don't cross concepts in their mind. Give you an example, made $1500 yday trading nvidia: I followed the curve for the last month, scalping a few times - I get a feel like panic point is ~180$, hype point ~195$, it's like that most swings. There were earnings yday, people are afraid that the company…

you apply it to find cross correlation ideas about larger numbers of assets. Try doing your stuff on a daily base with more than 500 assets :-)

But it won't give me anything interesting though ! Like, would you trust it on an even higher scale ? It has no basis for its investment thesis, it's a word statistician, not a risk-weighted decision taker !
Post reply on HN