Given that LLMs can't even finish Pokemon Red, how would you expect they are able to trade futures?
Hey! That wasn't easy!
Exploring the limits of large language models as quant traders
21–30 of 103 posts
Re: Exploring the limits of large language models as quant traders
#22Cool experiment, but it’s nothing more than a random walk.
Re: Exploring the limits of large language models as quant traders
#23The limits of LLM's for systematic trading were and are extremely obvious to anybody with a basic understanding of either field. You may as well be flipping a coin.
I agree. Plus it's way too short a timeframe to evaluate any trading activity seriously.
But I still think the experiment is interesting because it gives us insight into how LLMs approach risk management, and what effects on that we can have with prompting.
Re: Exploring the limits of large language models as quant traders
#24Given that LLMs can't even finish Pokemon Red, how would you expect they are able to trade futures?
(Unless you're a marketer) It makes a lot more sense to build a benchmark before the capabilities are there.
Re: Exploring the limits of large language models as quant traders
#25Are language models really the best choice for this? Seems to me that the outcome would be near random because they are so poorly suited. Which might manifest as > We also found that the models were highly sensitive to seemingly trivial prompt changes
No, LLMs are not a good choice for this – as the results show! If I had to guess, they're experimenting with LLMs for publicity.
Re: Exploring the limits of large language models as quant traders
#26Super interesting! You can click the "live" link in the header to see how they performed over time. The (geometric) average result at the end seems to be that the LLMs are down 35 % from their initial capital – and they got there in just 96 model-days. That's a daily return of -0.6 %, or a yearly return of -81 %, i.e. practically wiping out the starting capital. Although I lack the maths to determine it numerically (…
Agreed, and I'd also love to see a baseline of human performance here, both of experienced quant traders and of fresh grads who know the theory but never did this sort of trading and aren't familiar with the crypto futures market.
Re: Exploring the limits of large language models as quant traders
#27LLM's can do language but not much else, not poker, not trading and definitely no intelligence
Re: Exploring the limits of large language models as quant traders
#28At the end of the day it all comes down to input data. There are a lot of things you can do to collect proprietary data to give you an edge.
Re: Exploring the limits of large language models as quant traders
#29Given that LLMs can't even finish Pokemon Red, how would you expect they are able to trade futures?
i always felt that emotions, instincts, fear, greed, courage, pain are elements of a self-aware conscious loop system that can't be replicated accurately in a digital system and that a seasoned successful traders realize and utilize that the activity is largely is a psychological one. I'm not talking about neutral plays where you can absorb market fluctuations in the short term to extract 1~2% a week but directional trades that almost all traders play (regardless of how what exotic option strategies they are employing).
also the other curious nature of the markets is its ability to destroy any persistent trading system by reverting to its core stochastic properties and its constant ebb and flow from stability to instability that crescendos into systematic instability that rewrite the rules all over again.
ive tried all sorts of ways to do this and without being a large institution and being able to absorb the noise for neutral or legal quasi insider trading via proximity, for the average joe the emotional/psychological hardness you need to survive and be in the rather i think to myself the best trade is the simplest one: buy shares or invest in a business with money or time (strongly recommend against using this unless you have no other means) and sell it at a higher price or maintain a long term DCF from a business you own as leverage/collateral to arbitrage whatever rate your central bank sets on assets in demand or will be in demand.
to me its clear where LLM fits and doesn't but ultimately it cannot, will not, must not replace your own agency.
Re: Exploring the limits of large language models as quant traders
#30The limits of LLM's for systematic trading were and are extremely obvious to anybody with a basic understanding of either field. You may as well be flipping a coin.
20 years ago NNs were considered toys and it was "extremely obvious" to CS professors that AI can't be made to reliably distinguish between arbitrary photos of cats and dogs. But then in 2007 Microsoft released Asirra as a captcha problem [0], which prompted research, and we had an AI solving it not that long after.
Edit - additional detail: The original Asirra paper from October 2007 claimed "Barring a major advance in machine vision, we expect computers will have no better than a 1/54,000 chance of solving it" [0]. It took Philippe Golle from Palo Alto a bit under a year to get "a classifier which is 82.7% accurate in telling apart the images of cats and dogs used in Asirra" and "solve a 12-image Asirra challenge automatically with probability 10.3%" [1].
Edit 2: History is chock-full of examples of human ingenuity solving problems for very little external gain. And here we have a problem where the incentive is almost literally a money printing machine. I expect progress to be very rapid.
[0] https://www.microsoft.com/en-us/research/publication/asirra-...