Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reading SEC filings using LLMs

beatandraise.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Reading SEC filings using LLMs

#41

Post by ZephyrBlu »

nnechm wrote: post #31
ZephyrBlu wrote: post #27

Earlier quoted context omitted.

Looks cool and seems like a valuable tool! I really like the idea of LLMs that give you rich answers like this. I'm curious how you're accurately extracting the data though. Are you prompting to respond in a JSON format, using OpenAI's functions or something else? How do you ensure you have the correct label, dates, values, etc?

I am using openai's functions as that is a more reliable form of extracting json, but even that can fail sometimes as the response misses a "'" or a }

Nice. How often do functions fail? I haven't played around with them yet so no idea about reliability.

In regards to extracting the correct data, is that done through the function definitions? You specify all the fields you want to extract in the function and then let GPT go ham?

E.g. operating income, interest income, interest expense, etc.

Re: Reading SEC filings using LLMs

#42

Post by nnechm »

epups wrote: post #40

What about hallucinations though? If the LLM missed just one number, which at this point is very likely, you would get totally distorted information. Sorry if I missed something, but skimming through the site I didn't find any information about this specific issue.

As the answers are from the text, we can avoid hallucinations for the most part. I have not experienced made up numbers, instead errors are numbers that are typically misplaced, where it gives you revenues for the last 6 months instead of the last 3 months, when they are both next to each other in a table and so on... or the same numbers for different quarters and so on.

From my experience, GPT4 has been very good in following instructions and doesn't make up numbers which Bard is much more susceptible to. Bard relies heavily on snippets from the web search and completes the rest...

I am pretty sure if google wanted to train it, it would get all the answers right, but the way its designed, it gets one or two numbers and makes the rest up rather hilariously :) and even adds a breakdown which is also made up..

You can take a look here : https://imgur.com/a/vDxOV9D

Re: Reading SEC filings using LLMs

#43

Post by nnechm »

ZephyrBlu wrote: post #41
nnechm wrote: post #31

Earlier quoted context omitted.

I am using openai's functions as that is a more reliable form of extracting json, but even that can fail sometimes as the response misses a "'" or a }

Nice. How often do functions fail? I haven't played around with them yet so no idea about reliability. In regards to extracting the correct data, is that done through the function definitions? You specify all the fields you want to extract in the function and then let GPT go ham? E.g. operating income, interest income, interest expense, etc.

The functions tend to fail when the prompt is complex and the user asks for a lot of fields, and typically the last field in the json is not closed i.e missing a }, i guess openai is aware of it. It doesn't fail that often to have to write a workaround, atleast not yet.

So like a lot of applications, the problem boils down to being able to serve the right text. You have something that can read and do basic inference .... You need to tell it what to read so that it can answer your question. But it can only read 16k tokens (20k words at best). So that's the basic problem. As it's universal, i.e a problem across applications, its going to get better and information will be a lot easier to get access to...

Re: Reading SEC filings using LLMs

#45

Post by infecto »

refurb wrote: post #26

I'm pretty sure a primitive version of this has been done for a long time by hedge funds. You basically monitor the SEC website for when filings are made public, quickly parse the document in some automated way and make trades based on the output. It wouldn't surprise me if some hedge fund has been trying to build something with LLMs for past few years (and possibly already deployed it).

These already exist and are not primitive. Groups have been applying NLP and analysis against text for longer than you would think.

Re: Reading SEC filings using LLMs

#46

Post by nnechm »

nnechm wrote: post #6
drited wrote: post #4

Earlier quoted context omitted.

One of the advantages of data from CapIQ /Refinitiv is that you're not just pulling data from a single report but rather data has been curated across time from multiple historical reports so that historical income statements, balance sheets, footnote data etc spanning many years can be generated. When you say that generating graphs of gross margin, EV/Ebitda is just one step further, are you talking about generating…

I am talking about comparing multiple reports, i.e gross margin trends over 10 years and EV/TTM Ebitda etc across several reports. Currently only financials are possible, but the ratios depend on stock prices, so we are working on that ! You can think of it like this, you now have an army of readers that can go through tables really quickly. FactSet, CapIQ etc use a combination of automation/manual entry and fit thes…

It would be possible for example to get AMD's revenues over time like this, it's tedious because of context size, but it's unrestricted, so you can get whatever datapoint you want...

If we run this query over an api for income statements for all 8k companies, we pretty much have all income, balance sheet, cashflow items, shares outstanding etc. Add stock price data, that can give you EV/Ebitdas, P/Es and all that stuff.

https://imgur.com/a/2U2wt4h

Re: Reading SEC filings using LLMs

#47

Post by Terretta »

nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

This product would be a fantastic honeypot for front running researcher interest in particular investments.

I don't see anything in your privacy policy that gives me comfort my (even anonymized) interest in a particular firm isn't feeding your own signals.

Even if your policy promised it, I'd want to see technical controls implemented, since incentive to leverage information on searching would be so high.

Re: Reading SEC filings using LLMs

#48

Post by nnechm »

Terretta wrote: post #47
nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

This product would be a fantastic honeypot for front running researcher interest in particular investments. I don't see anything in your privacy policy that gives me comfort my (even anonymized) interest in a particular firm isn't feeding your own signals. Even if your policy promised it, I'd want to see technical controls implemented, since incentive to leverage information on searching would be so high.

Hi, Thanks for bringing that concern up. I shall keep it in mind and change it based on feedback from customers if it is an issue. Typically, people search for a company after a price move rather than before. And these are searches on publicly available data, ie data that is not proprietary and already filed with the sec. Needless to say, none of the chat traffic is used or will ever be used to feed any trading signals for anyone.

Re: Reading SEC filings using LLMs

#49

Post by alxdrlitreev »

We've recently made COFIN AI — an AI based on ChatGPT that reads hundreds of pages of SEC filings such as 10Ks and 10Qs and checks investor calls to assist you with investment research and strategy evaluation. Pretty much similar (almost the same) to something you did.

Check this out: https://cofinapp.com Let me know what you think!

Re: Reading SEC filings using LLMs

#50

Post by nnechm »

infecto wrote: post #45
refurb wrote: post #26

I'm pretty sure a primitive version of this has been done for a long time by hedge funds. You basically monitor the SEC website for when filings are made public, quickly parse the document in some automated way and make trades based on the output. It wouldn't surprise me if some hedge fund has been trying to build something with LLMs for past few years (and possibly already deployed it).

These already exist and are not primitive. Groups have been applying NLP and analysis against text for longer than you would think.

Yes, they are definitely not primitive, but considering how the breakthrough in LLMs happened, i.e MSFT, GOOGL were working on it, even as late as 2021, Google's BERT was not really there (at least the one they showed the public in the blog from Google research). So the sudden jump from ok to great thanks to openAI would probably mean a lot of existing systems also need to do that.
Post reply on HN