Given that LLMs can't even finish Pokemon Red, how would you expect they are able to trade futures?
Because trading is mainly number-based, unlike Pokemon Red?
Exploring the limits of large language models as quant traders
41–50 of 103 posts
Re: Exploring the limits of large language models as quant traders
#42Earlier quoted context omitted.
20 years ago NNs were considered toys and it was "extremely obvious" to CS professors that AI can't be made to reliably distinguish between arbitrary photos of cats and dogs. But then in 2007 Microsoft released Asirra as a captcha problem [0], which prompted research, and we had an AI solving it not that long after. Edit - additional detail: The original Asirra paper from October 2007 claimed "Barring a major advance…
What makes trading such a special case is that as you use new technology to increase the capability of your trading system, other market participants you are trading against will be doing the same; it's a never-ending arms race.
That doesn't mean it doesn't work. That means it does work!
If other market participants chose not to use something then that would show that it doesn't work.
Re: Exploring the limits of large language models as quant traders
#43Crazy how people continue to treat LLMs like they’re anything more than a record of past human knowledge and are then surprised when they can’t predict the future.
Re: Exploring the limits of large language models as quant traders
#44Given that LLMs can't even finish Pokemon Red, how would you expect they are able to trade futures?
Because trading is mainly number-based, unlike Pokemon Red?
I'll bite: What part of the game, which is encoded entirely by a finite set of numbers, takes input as numbers, provides output as numbers, and is processed by a CPU that acts in a discrete digital space, cannot be represented by numbers?
Re: Exploring the limits of large language models as quant traders
#45Earlier quoted context omitted.
This is true for some classes of strategies. At the same time there are strategies that can be profitable on longer timeframes. The two worlds are not mutually exclusive.
Yes, but LLM can barely cope with following the ordering of complex software tutorials linearly. Why would you reasonably expect them unprompted to understand time any better enough to trade and turn a profit?
My comment makes no such claim. I wrote about different timeframes that trading strategies operate on.
Re: Exploring the limits of large language models as quant traders
#46Are language models really the best choice for this? Seems to me that the outcome would be near random because they are so poorly suited. Which might manifest as > We also found that the models were highly sensitive to seemingly trivial prompt changes
No, LLMs are not a good choice for this – as the results show! If I had to guess, they're experimenting with LLMs for publicity.
Exactly. This is a performance by a really bad method actor.
Re: Exploring the limits of large language models as quant traders
#47Today it's clear that there are limitations to LLM's.
But I also see this incredible growth curve to LLM's improvement. 2 years ago, I wouldn't expect llm's to one shot a web application or help me debug obscure bugs and 2 years later I've been proven wrong.
I completely believe that trading is going to be saturated with ai traders in the future. And being able to predict and detect ai trading patterns is going to be an important leverage for human traders if they'll still exist
Re: Exploring the limits of large language models as quant traders
#48. . . "The (geometric) average result at the end seems to be that the LLMs are down 35 % from their initial capital – and they got there in just 96 model-days. That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital."
Proves that LLM's are nowhere near close to AGI.
Re: Exploring the limits of large language models as quant traders
#49Earlier quoted context omitted.
20 years ago NNs were considered toys and it was "extremely obvious" to CS professors that AI can't be made to reliably distinguish between arbitrary photos of cats and dogs. But then in 2007 Microsoft released Asirra as a captcha problem [0], which prompted research, and we had an AI solving it not that long after. Edit - additional detail: The original Asirra paper from October 2007 claimed "Barring a major advance…
The Asirra paper isn't from a ML research group. The statement: "Barring a major advance in machine vision, we expect computers will have no better than a 1/54,000 chance of solving it" is just a statement of fact - it wasn't any forms of prediction. If you read the paper you note that they surveyed researchers about the current state of the art ("Based on a survey of machine vision literature and vision ex- perts at…
Maybe I misunderstand, but it seems that there's nothing in your comment that contradicts any aspect of mine.
Regarding image classification. As I see it, a company like Microsoft surveying researchers about the state of the art and then making a business call to recommend the use of it as a captcha is significantly more meaningful of a prediction than any single paper from an ML research group. My intent was just to demonstrate that it was widely considered to be a significant open problem, which it clearly was. That in turn led to wider interest in solving it, and it was solved soon after - much faster than expected by people I spoke to around that time.
Regarding stock market prediction, of course I'm not claiming that long term prediction is possible. All I'm saying is that I don't see a reason why quant trading could be used as a captcha - it's as pure a pattern matching task as could be, and if AIs can employ all the context and tooling used by humans, I would expect them to be at least as good as humans within a few years. So my prediction is not the end of quant trading, but rather that much of the work of quants would be overtaken by AIs.
Obviously a big part of trading at the moment is already being done by AIs, so I'm not making a particularly bold claim here. What I'm predicting (and I don't believe that anyone in the field would actually disagree) is that as tech advances, AIs will be given control of longer trading time horizons, moving from the current focus on HFT to day trading and then to longer term investment decisions. I believe that there will still be humans in the loop for many many years, but that these humans would gradually turn their focus to high level investment strategy rather than individual trades.
Re: Exploring the limits of large language models as quant traders
#50I was chatting to a friend in the space. This guy is both experienced in trading and LLMs, and has gone all-in on using LLMs to get his day-to-day coding done. Now he's working on the model to end all models, which is a fairly ambitious way to put it, but it throws off some interesting conversations. You need domain knowledge to get this to work. Things like "we fed the model the market data" are actually non-obvious…
[dead]