Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploring the limits of large language models as quant traders

nof1.ai

Post reply on HN
Print view View original

71–80 of 103 posts

Re: Exploring the limits of large language models as quant traders

#71

Post by falcor84 »

nl wrote: post #60
falcor84 wrote: post #49

Earlier quoted context omitted.

Maybe I misunderstand, but it seems that there's nothing in your comment that contradicts any aspect of mine. Regarding image classification. As I see it, a company like Microsoft surveying researchers about the state of the art and then making a business call to recommend the use of it as a captcha is significantly more meaningful of a prediction than any single paper from an ML research group. My intent was just to…

> making a business call to recommend the use of it as a captcha is significantly more meaningful of a prediction than any single paper from an ML research group. That's not what this is. It's a research paper from 3 researchers at MSR.

Ok, I'll take it. It definitely wasn't a business call at the level of Microsoft saying that everyone should be using it, but it was an actual service offered under the Microsoft umbrella and used by many sites in the wild, e.g. via this MediaWiki extension [0], for 8 years [1].

[0] https://www.mediawiki.org/wiki/Extension:Asirra

[1] https://web.archive.org/web/20150207180225/https%3A//researc...

Re: Exploring the limits of large language models as quant traders

#72

Post by callamdelaney »

lambdaone wrote: post #38
falcor84 wrote: post #30

Earlier quoted context omitted.

20 years ago NNs were considered toys and it was "extremely obvious" to CS professors that AI can't be made to reliably distinguish between arbitrary photos of cats and dogs. But then in 2007 Microsoft released Asirra as a captcha problem [0], which prompted research, and we had an AI solving it not that long after. Edit - additional detail: The original Asirra paper from October 2007 claimed "Barring a major advance…

What makes trading such a special case is that as you use new technology to increase the capability of your trading system, other market participants you are trading against will be doing the same; it's a never-ending arms race.

The only applications of generative AI I can envisage for trading, systematically or otherwise are the following:

  - data extraction: It's possible to get pretty good levels of accuracy on unstructured data, eg financial reports with relatively little effort compared to before decent llm's
   - sentiment analysis: Why bother with complicated sentiment analysis when you can just feed an article into an LLM for scoring?
   - reports: You could use it to generate reports on your financial performance, current positions etc
   - code: It can generate some code that might sometimes be useful in the development of a system
The issue is that these models don't really reason and they trade in what might as well be a random way. For example, a stock might have just dropped 5%. One LLM might say that we should buy the stock now and follow a mean reversion strategy. Another may say we should short the stock and follow the trend. The same LLM may give the same output on a different call. A miniscule difference in price, time or other data will potentially change the output when really a signal should be relatively robust.

And if you're going to tell the model say, 'we want to look for mean reversion opportunities' - then why bother with an LLM?

Another angle: LLM's are trained on the vast swathe of scammy internet content and rubbish in relation to the stock market. 90%+ of active retail traders lose money. If an llm is fed on losing / scammy rubbish, how could it possibly produce a return?

Re: Exploring the limits of large language models as quant traders

#73

Post by red-iron-pine »

Saline9515 wrote: post #39
callamdelaney wrote: post #12

The limits of LLM's for systematic trading were and are extremely obvious to anybody with a basic understanding of either field. You may as well be flipping a coin.

So what are the limits, given that you seem knowledgeable about it?

they're language models. they exist to take in text and compare it to existing tokens.

they're not quant-bots that already exist to read in stock prices and make decisions. different kind of ML/AI

from TFA: "We also found that the models were highly sensitive to seemingly trivial prompt changes"

Re: Exploring the limits of large language models as quant traders

#74

Post by michaelcampbell »

kqr wrote: post #2

Super interesting! You can click the "live" link in the header to see how they performed over time. The (geometric) average result at the end seems to be that the LLMs are down 35 % from their initial capital – and they got there in just 96 model-days. That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital. Although I lack the maths to determine it numerically (…

LLM's know the WORDS of "The market can remain irrational longer than you can remain solvent", but not the meaning.

Re: Exploring the limits of large language models as quant traders

#75

Post by esseph »

terminalbraid wrote: post #44
Saline9515 wrote: post #41

Earlier quoted context omitted.

Because trading is mainly number-based, unlike Pokemon Red?

I'll bite: What part of the game, which is encoded entirely by a finite set of numbers, takes input as numbers, provides output as numbers, and is processed by a CPU that acts in a discrete digital space, cannot be represented by numbers?

The joy the player experiences in playing the game and becoming the best Pokemon Trainer :)

Re: Exploring the limits of large language models as quant traders

#76

Post by eru »

aswegs8 wrote: post #13

Given that LLMs can't even finish Pokemon Red, how would you expect they are able to trade futures?

Computers managed to beat the world's best humans at chess long before they managed how learn how to walk. What's easy and hard for humans isn't necessarily a good guide to what's easy or hard for a computer.

(And I'm fairly sure it would be pretty easy to build a system that uses an LLM and a few other small components to beat Pokemon Red. The experiment you are talking about is deliberately hobbled by using a stock LLM without any such tools to make the whole thing entertaining. But when you are trading, you'd want to give your LLM as much help as possible.)

Re: Exploring the limits of large language models as quant traders

#77

Post by falcor84 »

callamdelaney wrote: post #72
lambdaone wrote: post #38

Earlier quoted context omitted.

What makes trading such a special case is that as you use new technology to increase the capability of your trading system, other market participants you are trading against will be doing the same; it's a never-ending arms race.

The only applications of generative AI I can envisage for trading, systematically or otherwise are the following: - data extraction: It's possible to get pretty good levels of accuracy on unstructured data, eg financial reports with relatively little effort compared to before decent llm's - sentiment analysis: Why bother with complicated sentiment analysis when you can just feed an article into an LLM for scoring? -…

> If an llm is fed on losing / scammy rubbish, how could it possibly produce a return?

Rather than just relying on pretraining, you'd use RL on the trade outcomes.

Re: Exploring the limits of large language models as quant traders

#78

Post by Lapsa »

ritonlajoie wrote: post #34
Lapsa wrote: post #20

Earlier quoted context omitted.

I would argue that sentiment classification is where LLMs perform best. folks are already using it for precisely such purpose - have even built a public index out of it

what index ?

sorry dude. tried going down the rabbit hole but I'm too lazy and uninterested in it. read about it month ago or so. perhaps Daily News Sentiment Index uses LLMs, not sure. if you go long enough through https://quantocracy.com/ you should be able to find it

Re: Exploring the limits of large language models as quant traders

#79

Post by spaceman_2020 »

kqr wrote: post #59
spaceman_2020 wrote: post #53

Earlier quoted context omitted.

As someone who trades crypto semi-professionally, this was one of the toughest trading periods I've ever seen and included a massive liquidation event on 10th of October that wiped out over $20B in capital. Any trader who broke even in this period likely outperformed. I know some very, very good traders who got wiped out on leverage on 10th of October when stop losses didn't trigger and prices plummetted to 2021 leve…

Note that 10th of October is before the trading period in this experiment. If anything, autoregression over shorter timescales would suggest entering after 10th of October being a good idea!

Just noticed that it was after October 10th.

I'll still say that the trading period after October 10th has been brutally choppy. Only now do we have a clear direction (down) where you can at least short with some confidence

Re: Exploring the limits of large language models as quant traders

#80

Post by ponector »

kqr wrote: post #2

Super interesting! You can click the "live" link in the header to see how they performed over time. The (geometric) average result at the end seems to be that the LLMs are down 35 % from their initial capital – and they got there in just 96 model-days. That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital. Although I lack the maths to determine it numerically (…

>> That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital

LLM indeed can replace average human being.

Post reply on HN