51–60 of 93 posts
I’ve had pretty dismal results doing the same with spreadsheets—even with the data nicely tagged (and numbers directly adjacent to the labels) GPT-4o would completely make up figures to satisfy the JSON schema passed to it. YMMV.
Earlier quoted context omitted.
I usually come to a different conclusion using the JSON output on Lamini, e.g. even with Llama 3.2 3B https://lamini-ai.github.io/inference/json_output Most of these models can read. If the relevant facts are in the prompt, they can almost always extract them correctly. Of course bigger models do better on more complex tasks and reasoning unless you use finetuning or memory tuning.
You should probably disclose you're the founder of lamini. Do you have any publicly available validation data demonstrating 100% json compliance?
Obviously I’m biased, but I also spend every day using tools like this.
Regarding json compliance, we have a formal grammar and a test suite. If you find a bug please report it. I’d appreciate having more test coverage.
We built this huge system with tons of regexes, custom parsers, word lists, ontologies etc. It was a huge effort to get somewhat acceptable accuracy.
It is humbling to see that these days a 100 line Python script can do the same thing but better: AI has basically taken over my first job.
I wonder, did anyone try to fine-tune a model specifically for general formatted data extraction? My naive thinking is that this should be pretty doable - after all, it's basically just restructuring the content using mostly the same tokens as input.
The reason why this would be useful (in my case) is because while large LLMs are perfectly capable of extraction, I often need to run it on millions of texts, which would be too costly. That's the reason I usually end up creating a custom small model, which is faster and cheaper. But a general small extraction-focused LLM would solve this.
I thought about fine-tuning Llama3-1B or Qwen models on larger models outputs, but my focus is currently elsewhere.
What a sad state for humanity that we have to resort to this sort of OCR/scrapping instead of the original data being released in a machine readable format in the first place.
We used GPT 4o for more or less the same stuff. Got a boatload of scanned bills we had to digitize, and GPT really nailed the task. Made a schema, and just fed the model all the bills. Worked better than any OCR we tried.
How are you going to find (not even talking about correcting) hallucinated errors? If money is involved and the LLM produces hallucination errors, how do you handle monetary impacts of such errors? How does that approach scale financially?
The images are tables with 4 columns and 10 rows of numbers and metadata above that are in a couple of fields. We had thousands of images already loaded and when we tried to check those previously loaded images we found quite a few errors.
My first job (around 2010) was to extract events from financial news and police reports. We built this huge system with tons of regexes, custom parsers, word lists, ontologies etc. It was a huge effort to get somewhat acceptable accuracy. It is humbling to see that these days a 100 line Python script can do the same thing but better: AI has basically taken over my first job.
The SEC's EDGAR database (which is for SEC filings) is another nightmare ready to end. Extracting individual sections from a filing is, afaik, impossible pragmatically.
I tried making two parsers: https://github.com/MegaManSec/SEC-Feed-Parser and https://github.com/MegaManSec/SEC-sec-incident-notifier but they're just hacks.
Then just link it up to your automated investment platform and you're ready to go!