Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reading SEC filings using LLMs

beatandraise.com

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Reading SEC filings using LLMs

#21

Post by nnechm »

halotrope wrote: post #20

This looks great! We have a similar tool on https://markets.sh but more geared towards news and general information about stocks (You can ask e.g "What are the current issues with the Disney company?") We found the hardest part about this is the pre-ranking of sections and finding the relevant information. Do you feed the whole report into OpenAI?

Thanks, I don't feed the whole report in, unless necessary. Context size is a real limiting factor :), so you need to split it up into chunks that are meaningful on their own and find those relevant to the question. In this case, what are the current issues with the Disney company, This is how I would do it :) I get all the news articles related to Disney, this would then have sections (depends if you are using a json or a scraper), now you would then need to pick only those chunks and the surrounding ones (this would be an on the fly search) and then pass it to openai... This one is a very open ended question, this is what I get in beatandraise.com :) https://imgur.com/a/dqmcrRA

Re: Reading SEC filings using LLMs

#22

Post by dcsan »

anotherhue wrote: post #3
nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

I've been trying something similar with parliamentary debates. They're long winded, often full of empty speech, and a chore to read. The LLMs are able to hone in on the details and provide interesting responses like "What questions were asked of the minister that they failed to address" and "What should the opposition leader have mentioned in their response that the minister would have found difficult to answer" Cruc…

Could you ask an LLM if they were persuaded by a speaker of one side of a debate as a method of evaluation? Ie the bot's before and after opinion based on fine-tuning with the pro and con arguments?

I was also thinking about a society of bots type application where you could have autonomous bot researchers, debaters, judges and audience. Would be interesting to feed in the topics and grab some popcorn

Re: Reading SEC filings using LLMs

#23

Post by nnechm »

nnechm wrote: post #21
halotrope wrote: post #20

This looks great! We have a similar tool on https://markets.sh but more geared towards news and general information about stocks (You can ask e.g "What are the current issues with the Disney company?") We found the hardest part about this is the pre-ranking of sections and finding the relevant information. Do you feed the whole report into OpenAI?

Thanks, I don't feed the whole report in, unless necessary. Context size is a real limiting factor :), so you need to split it up into chunks that are meaningful on their own and find those relevant to the question. In this case, what are the current issues with the Disney company, This is how I would do it :) I get all the news articles related to Disney, this would then have sections (depends if you are using a jso…

I don't do handle news articles, and was just focusing on filings, so my answer would look for a filing. That's why it looks the way it does :)

Re: Reading SEC filings using LLMs

#24

Post by andylynch »

nnechm wrote: post #9
cerved wrote: post #5

Earlier quoted context omitted.

Neat idea! How do you parse the PDFs?

Pdf parsing was more tedious that I would have liked at this stage so I stuck to the SEC which requires that companies file in a text format :) so that helped. I used poppler on a digital ocean droplet, but the sheer variety of company pdfs especially european companies, some of which have to be OCRed, meant results were not really uniform. GPT still does very well, but not as well as on text documents directly. So i…

Are you pulling out the inline XBRL or taking a different approach? I'm curious to hear others' practical experience with it good or bad.

Re: Reading SEC filings using LLMs

#25

Post by nnechm »

andylynch wrote: post #24
nnechm wrote: post #9

Earlier quoted context omitted.

Pdf parsing was more tedious that I would have liked at this stage so I stuck to the SEC which requires that companies file in a text format :) so that helped. I used poppler on a digital ocean droplet, but the sheer variety of company pdfs especially european companies, some of which have to be OCRed, meant results were not really uniform. GPT still does very well, but not as well as on text documents directly. So i…

Are you pulling out the inline XBRL or taking a different approach? I'm curious to hear others' practical experience with it good or bad.

I am not using inline XBRL, instead relying on gpt's ability to read tables.

Re: Reading SEC filings using LLMs

#26

Post by refurb »

I'm pretty sure a primitive version of this has been done for a long time by hedge funds.

You basically monitor the SEC website for when filings are made public, quickly parse the document in some automated way and make trades based on the output.

It wouldn't surprise me if some hedge fund has been trying to build something with LLMs for past few years (and possibly already deployed it).

Re: Reading SEC filings using LLMs

#27

Post by ZephyrBlu »

nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

Looks cool and seems like a valuable tool! I really like the idea of LLMs that give you rich answers like this.

I'm curious how you're accurately extracting the data though. Are you prompting to respond in a JSON format, using OpenAI's functions or something else? How do you ensure you have the correct label, dates, values, etc?

Re: Reading SEC filings using LLMs

#28

Post by nnechm »

refurb wrote: post #26

I'm pretty sure a primitive version of this has been done for a long time by hedge funds. You basically monitor the SEC website for when filings are made public, quickly parse the document in some automated way and make trades based on the output. It wouldn't surprise me if some hedge fund has been trying to build something with LLMs for past few years (and possibly already deployed it).

You are probably right, but there has been nothing like gpt4 before, having seen some versions before I would wager that their entire application can probably be redone for a fraction of the cost and it would work better... :)

Re: Reading SEC filings using LLMs

#29

Post by blawson »

nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

I tried this as a little hobby weekend project but found that after a while it would start hallucinating answers even if previously it had gotten them right. It didn’t even take that long sometimes, where I’d ask a question about revenue, then liabilities, and them to sum some revenue numbers and they would just start to be wrong.

I wouldn’t yet feel comfortable with this without some automated reconciliation which to my mind defeated the point of my hobby project but I’m curious if you’ve seen different? No doubt you’d expect this to improve over time though.

Re: Reading SEC filings using LLMs

#30

Post by nnechm »

blawson wrote: post #29
nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

I tried this as a little hobby weekend project but found that after a while it would start hallucinating answers even if previously it had gotten them right. It didn’t even take that long sometimes, where I’d ask a question about revenue, then liabilities, and them to sum some revenue numbers and they would just start to be wrong. I wouldn’t yet feel comfortable with this without some automated reconciliation which t…

You can try it out for yourself... :) Here's an example, that asks for AMD's cash and makes an arbitrary calculation on total liabilities, the ai is smart enough to sum up everything until equity and gets the numbers right, without any hallucination.

https://imgur.com/a/oAUZiIB

This is the source: https://www.sec.gov/Archives/edgar/data/2488/000000248823000...

The sum of all of these is 12831.

Total Current Liabilities 7572 Long-term debt, net of current portion 1714 Long-term operating lease liabilities 393 Deferred tax liabilities 1365 Other long-term liabilities 1787 12831

Post reply on HN