Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reading SEC filings using LLMs

beatandraise.com

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Reading SEC filings using LLMs

#31

Post by nnechm »

ZephyrBlu wrote: post #27
nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

Looks cool and seems like a valuable tool! I really like the idea of LLMs that give you rich answers like this. I'm curious how you're accurately extracting the data though. Are you prompting to respond in a JSON format, using OpenAI's functions or something else? How do you ensure you have the correct label, dates, values, etc?

I am using openai's functions as that is a more reliable form of extracting json, but even that can fail sometimes as the response misses a "'" or a }

Re: Reading SEC filings using LLMs

#32

Post by sroussey »

I am working on something similar. I have all the 10-k and 8-k docs.

I’ve pulled out the structured data separately, and now looking at breaking up the text into paragraphs to get embeddings.

Why are you using inverted index style text search instead of embeddings? I can see doing both perhaps…

Re: Reading SEC filings using LLMs

#33

Post by paulkon »

nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

Could you share more about the prompt you use to get GPT-4 to return a query from the user's question? This is fed into postgres full text search?

Re: Reading SEC filings using LLMs

#34

Post by nnechm »

sroussey wrote: post #32

I am working on something similar. I have all the 10-k and 8-k docs. I’ve pulled out the structured data separately, and now looking at breaking up the text into paragraphs to get embeddings. Why are you using inverted index style text search instead of embeddings? I can see doing both perhaps…

Embeddings did not really work well enough for me, let us say you are looking for numerical text, so for e,g get me AMD's revenue from March 2022. The embedding representation needs to understand that March 2022 together is way more important than March alone, I often ended up with March 2021 or March 2018 as being closer because the text might have multiple terms containing 'revenue' or multiple 'march'... Perhaps I could have improved it, but that did not seem like the right path to go down for accuracy.. This was way worse for e.g when I am looking for ECB statements, they can refer to an older date in their current report and it caused all sorts of trouble :). An initial fix was to basically mention March several times so that the search returns that... :)

Re: Reading SEC filings using LLMs

#35

Post by nnechm »

paulkon wrote: post #33
nnechm wrote: post #2

I have been working on getting ChatGPT to answer questions that equity research analysts, investors would like to get from SEC filings. The application uses a combination of hybrid text search and LLMs for completion and does not rely much on embedding based distance searches. A core assumption underlying this is that LLMs are already pretty good and will continue to get better at reading texts. If provided with the…

Could you share more about the prompt you use to get GPT-4 to return a query from the user's question? This is fed into postgres full text search?

I am using openai's latest functions api, so you can get it to return arguments that will ensure that you get a json, it works pretty well most of the time. The json would then be used to fetch a report from a database.

Re: Reading SEC filings using LLMs

#36

Post by xcxcx »

refurb wrote: post #26

I'm pretty sure a primitive version of this has been done for a long time by hedge funds. You basically monitor the SEC website for when filings are made public, quickly parse the document in some automated way and make trades based on the output. It wouldn't surprise me if some hedge fund has been trying to build something with LLMs for past few years (and possibly already deployed it).

This was done by hand by people like Footnoted. They received SEC filings using a key red flag word filter into a shared Gmail account with special attention for filings done on Friday night or ahead of the holidays.

I think using LLMs to ask a company about their company figures is missing the point. After all, XBRL already exists, and that data is widely accessible anyways. Where LLMs would add more value is analyzing changes in different sections of a company's filings, such as Risk Factors, MD&A, or governance filings.

Re: Reading SEC filings using LLMs

#37

Post by nnechm »

xcxcx wrote: post #36
refurb wrote: post #26

I'm pretty sure a primitive version of this has been done for a long time by hedge funds. You basically monitor the SEC website for when filings are made public, quickly parse the document in some automated way and make trades based on the output. It wouldn't surprise me if some hedge fund has been trying to build something with LLMs for past few years (and possibly already deployed it).

This was done by hand by people like Footnoted. They received SEC filings using a key red flag word filter into a shared Gmail account with special attention for filings done on Friday night or ahead of the holidays. I think using LLMs to ask a company about their company figures is missing the point. After all, XBRL already exists, and that data is widely accessible anyways. Where LLMs would add more value is analyz…

Yes, I think the broader point is you have these readers available on call via api, so you can do all that you suggested and more...

Re: Reading SEC filings using LLMs

#39

Post by notahacker »

dcsan wrote: post #22
anotherhue wrote: post #3

Earlier quoted context omitted.

I've been trying something similar with parliamentary debates. They're long winded, often full of empty speech, and a chore to read. The LLMs are able to hone in on the details and provide interesting responses like "What questions were asked of the minister that they failed to address" and "What should the opposition leader have mentioned in their response that the minister would have found difficult to answer" Cruc…

Could you ask an LLM if they were persuaded by a speaker of one side of a debate as a method of evaluation? Ie the bot's before and after opinion based on fine-tuning with the pro and con arguments? I was also thinking about a society of bots type application where you could have autonomous bot researchers, debaters, judges and audience. Would be interesting to feed in the topics and grab some popcorn

I think you could ask an LLM if they were persuaded, but I don't think you'd get meaningful data from it

Leaving aside the bots that are trained to answer "as an LLM I do not have opinions..." it's going to be a very basic probabilistic yes or no based on tone and numbers of pros and cons rather than knowledge of the surrounding political context and higher order reasoning about the accuracy of the claimed pros and cons.

Re: Reading SEC filings using LLMs

#40

Post by epups »

What about hallucinations though? If the LLM missed just one number, which at this point is very likely, you would get totally distorted information.

Sorry if I missed something, but skimming through the site I didn't find any information about this specific issue.

Post reply on HN