Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reading SEC filings using LLMs

beatandraise.com

Post reply on HN
Print view View original

11–20 of 78 posts

Re: Reading SEC filings using LLMs

#11

Post by lifeisstillgood »

nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

So (if I get it right) you are using the LLM to convert the human language question into (presumably) code that is basis for running "normal" searches and returning text.

Avoiding any fears of hallucinations.

But earlier you say "LLMs will get better at comprehension". So are you using an LLM to markup the original text in some way ?

Re: Reading SEC filings using LLMs

#12

Post by nnechm »

lifeisstillgood wrote: post #11
nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

So (if I get it right) you are using the LLM to convert the human language question into (presumably) code that is basis for running "normal" searches and returning text. Avoiding any fears of hallucinations. But earlier you say "LLMs will get better at comprehension". So are you using an LLM to markup the original text in some way ?

Yes, that's right, I avoid hallucinations that way. In addition, I mark up existing texts so that the LLM knows what it is reading even if it is a piece of a larger text, so for e.g if you need to get Apple's Greater China sales for each quarter in 2020... https://imgur.com/a/oCCay3o

I am not making 4 calls with the entire text, I instead get pieces of each which would best match the question. There are additional challenges, for instance, GPT4 struggles to know the difference between the words guidance and outlook, which mean the same thing but somehow they don't for GPT4.

When I say they become better readers, I meant it in a general sense as in better in the case above. You basically have someone who can read through tables really well, and that can change investment research fundamentally, which is a lot of reading tables and graphs :)

Re: Reading SEC filings using LLMs

#13

Post by anotherhue »

nnechm wrote: post #8
anotherhue wrote: post #3

Earlier quoted context omitted.

I've been trying something similar with parliamentary debates. They're long winded, often full of empty speech, and a chore to read. The LLMs are able to hone in on the details and provide interesting responses like "What questions were asked of the minister that they failed to address" and "What should the opposition leader have mentioned in their response that the minister would have found difficult to answer" Cruc…

I think one thing you can try is to figure out what lies where, so chunking arbitrarily will not work as well as chunking with headings for e.g For e.g, if the question is: "What should the opposition leader have mentioned in their response that the minister would have found difficult to answer." An embedding based search will find it fairly difficult to match this against a text. Based on my experience, you have to…

> "We're definitely moving forward with strong measures to improve the situation, and while progress might not be immediately visible, we're fully engaged in this essential journey."

Re: Reading SEC filings using LLMs

#14

Post by nnechm »

anotherhue wrote: post #13
nnechm wrote: post #8

Earlier quoted context omitted.

I think one thing you can try is to figure out what lies where, so chunking arbitrarily will not work as well as chunking with headings for e.g For e.g, if the question is: "What should the opposition leader have mentioned in their response that the minister would have found difficult to answer." An embedding based search will find it fairly difficult to match this against a text. Based on my experience, you have to…

> "We're definitely moving forward with strong measures to improve the situation, and while progress might not be immediately visible, we're fully engaged in this essential journey."

This is how GPT4 answers the question... I assumed the question, but presumably will be along those lines :)

https://chat.openai.com/share/c736abf4-ae3c-4fbd-9427-b7d2f9...

Re: Reading SEC filings using LLMs

#15

Post by hef19898 »

nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

I once had a collegue who used ChatGPT to sumarize our then employer's SEC filings. Results were, well, putting it mildly, mixed. Best case was a slightly less biased version of the "shareholder letters" (read: propaganda pieces) published around the same time.

What ChatGPT completey missed was stuff like omissions (I'll kind of give it a pass here, how can software analyse the absence of something without having access to supplemental documents; it still shoes hoe dangerous it is to rely only on a LLM for such analysis) and, more importantly, the connection between certain tid bits and, and here it became outright dangerous, ChatGPT didn't provide anything meaningful on risks and financials.

The tid bit it missed, one of the most important ones at the time, was a huge multi year contract given to a large investor in said company. To find it, including the honestly hilarious amount, one had to connect the disclosure of not specified contract to a named investor, the specifics of said contract (not mentioning the investor by name), the amount stated in some finacial statement from the document and, here obviously ChatGPT failed completely, knowledge of what said investor (a pretty (in)-famous company) specialized in. ChatGPT did even mention a single of those data points. Fun fact, said contract covered a significant junk of the amount the investing company had invested to begin with. And all that during a time in which the financial stability of the reporting company was at least questionable. Oh, and ChatGPT didn't even realize that risk (cash and equivilants on hand devided by burn rate per year is simple maths), or repeat the exact passage in which the SEC filling said that the survival of the reporting was in doubt.

In short, without some serious promp working, and including addditional data sources, I think ChatGPT is utterly useless in analyzing SEC filings, even worse it can be outright misleading. Not that SEC filings are increadibly hard to read, some basic financial knowledge and someone pointing out the highlights, based on a basic unserstanding of how those filongs actually work are supossed to work, and you are there.

Re: Reading SEC filings using LLMs

#16

Post by hef19898 »

anotherhue wrote: post #13
nnechm wrote: post #8

Earlier quoted context omitted.

I think one thing you can try is to figure out what lies where, so chunking arbitrarily will not work as well as chunking with headings for e.g For e.g, if the question is: "What should the opposition leader have mentioned in their response that the minister would have found difficult to answer." An embedding based search will find it fairly difficult to match this against a text. Based on my experience, you have to…

> "We're definitely moving forward with strong measures to improve the situation, and while progress might not be immediately visible, we're fully engaged in this essential journey."

Man, it seems the people writing politicians, or managers, statements seem at serious risk being replaced with a LLM...

Re: Reading SEC filings using LLMs

#17

Post by yawnxyz »

cerved wrote: post #5
nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

Neat idea! How do you parse the PDFs?

I have a tool that applies LLMs to abstracts and research papers — @opendocsg/pdf2md on Node / Sveltekit has been really good for me

Re: Reading SEC filings using LLMs

#18

Post by nnechm »

hef19898 wrote: post #15
nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

I once had a collegue who used ChatGPT to sumarize our then employer's SEC filings. Results were, well, putting it mildly, mixed. Best case was a slightly less biased version of the "shareholder letters" (read: propaganda pieces) published around the same time. What ChatGPT completey missed was stuff like omissions (I'll kind of give it a pass here, how can software analyse the absence of something without having acc…

I do not really imagine this as something that does all the investment work and makes a decision.

Instead, the mental model is you have an army of people who can read texts really well, as in 'reading comprehension' as they call it in english tests... This army can get you information on the fly.

Investment research involves a lot of back and forth reading and fetching tables and making conclusions, which in turn might not have much to do with stock price performance, but there's a whole industry of financial information and news for that :)

So currently, the scope is to make widely available information beyond what FactSet and CapIQ offer and even that's a long way away :)

Re: Reading SEC filings using LLMs

#19

Post by nnechm »

yawnxyz wrote: post #17
cerved wrote: post #5

Earlier quoted context omitted.

Neat idea! How do you parse the PDFs?

I have a tool that applies LLMs to abstracts and research papers — @opendocsg/pdf2md on Node / Sveltekit has been really good for me

Thanks ! It's really interesting to see how LLMs have now brought back projects that were last updated years ago back into the limelight :)

Re: Reading SEC filings using LLMs

#20

Post by halotrope »

This looks great! We have a similar tool on https://markets.sh but more geared towards news and general information about stocks (You can ask e.g "What are the current issues with the Disney company?")

We found the hardest part about this is the pre-ranking of sections and finding the relevant information. Do you feed the whole report into OpenAI?

Post reply on HN