Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: BioTradingArena – Benchmark for LLMs to predict biotech stock movements

biotradingarena.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Show HN: BioTradingArena – Benchmark for LLMs to predict biotech stock movements

#11

Post by mempko »

I'm curious about something. If this is based on historical datasets, and people build strategies using LLMs, then in theory this is deeply flawed since LLMs would contain the knowledge about some of the datasets, and certainly the prices of the biotech stocks. This approach cannot be used to figure out which strategies are good because they know the future outcome.

How do you prevent this problem? It's a classic problem in backtesting strategies where you leak future information into the model.

EDIT: Some context, I ran a quant fund before.

Re: Show HN: BioTradingArena – Benchmark for LLMs to predict biotech stock movements

#12

Post by dchu17 »

mempko wrote: post #11

I'm curious about something. If this is based on historical datasets, and people build strategies using LLMs, then in theory this is deeply flawed since LLMs would contain the knowledge about some of the datasets, and certainly the prices of the biotech stocks. This approach cannot be used to figure out which strategies are good because they know the future outcome. How do you prevent this problem? It's a classic pro…

Yes this is a major problem I thought about. The makeshift solution here was to redact the “identifying information” on the press release. Even then, I benchmarked that GPT-5 could still match it back to the right TIKR around 53% of the time. It does not seem to be able to recall the price of the stock in my benchmark, but to be honest I’m not entirely sure how trustworthy this benchmark is and I may need to come up with a few more clever solutions to validate.

One solution could be to get experts to write similar press releases so that the text itself is out of distribution or if an actual quant firm has internal models, they can just make sure that there is a cutoff date to the pre-training data.

I'm curious, when you ran a quant fund, what was your approach?

Re: Show HN: BioTradingArena – Benchmark for LLMs to predict biotech stock movements

#13

Post by mempko »

dchu17 wrote: post #12
mempko wrote: post #11

I'm curious about something. If this is based on historical datasets, and people build strategies using LLMs, then in theory this is deeply flawed since LLMs would contain the knowledge about some of the datasets, and certainly the prices of the biotech stocks. This approach cannot be used to figure out which strategies are good because they know the future outcome. How do you prevent this problem? It's a classic pro…

Yes this is a major problem I thought about. The makeshift solution here was to redact the “identifying information” on the press release. Even then, I benchmarked that GPT-5 could still match it back to the right TIKR around 53% of the time. It does not seem to be able to recall the price of the stock in my benchmark, but to be honest I’m not entirely sure how trustworthy this benchmark is and I may need to come up…

We didn't use other LLMs. We built our own models and had a system designed to never leak future information at any given timepoint. Models can only access data the system allowed it at any time point gating future information. This means even training has to use the same system.

You have to design it from the ground up with that approach. Just to give you an idea of how hard it is, when a company releases an earnings report, they can update it in the future with corrected information, so if you pull it later you will leak future information into the past. So even basics like earnings need to be versioned by time.

But you know, most people don't really care and think they have an edge, and who knows maybe they do. Only live trading will prove it.

Re: Show HN: BioTradingArena – Benchmark for LLMs to predict biotech stock movements

#16

Post by genes_unknown_1 »

dchu17 wrote: post #9
genes_unknown_1 wrote: post #8

I used to work at a private investment fund as a data engineer for building in house models to evaluate drug programs and biotech companies. We took a pretty varied approach with catalysts, investment data, people data, trial data, but also analyses on the molecule and drug itself. It was a lot of work and I really don't think we made a dent into understanding what succeeds and what doesnt. Also investors in biotech…

That's interesting. I am curious, what kind of analyses did you work with on the molecule and drug itself? Was it like mostly reading papers/patents or did your team do anything experimental?

Nothing experimental. Only using foundational models. And usually basic stuff thats more relevant to the indication we were looking ie does it cross the bbb or potency models
Post reply on HN