Earlier quoted context omitted.
This is spot on, any legacy vendor focusing on a specific type of PDF is going to get obliterated by LLMs. The problem with using an off-the-shelf provider like this is, you get stuck with their data schema. With an LLM, you have full control over the schema meaning you can parse and extract much more unique data. The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extra…
> The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extract the data we need, validate its performance, and deploy it with confidence into prod?" A smart vendor will shift into that space - they'll use that LLM themselves, and figure out some combination of finetunes, multiple LLMs, classical methods and human verification of random samples, that lets them not only "va…
Ingesting PDFs and why Gemini 2.0 changes everything
451–456 of 456 posts
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#4521) I don't mind destroying the binding to get the best quality. How do I do so?
2) I have a multipage double-sided scanner (fujitsu scansnap). would this be sufficient to do the scan portion?
3) Is there anything that determines the font of the book text and reproduces that somehow? and that deals with things like bold and italic and applies that either as markdown output or what have you?
4) how do you de-paginate the raw text to reflow into (say) an epub format that will paginate based on the output device specification?
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#453I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific ty…
This is spot on, any legacy vendor focusing on a specific type of PDF is going to get obliterated by LLMs. The problem with using an off-the-shelf provider like this is, you get stuck with their data schema. With an LLM, you have full control over the schema meaning you can parse and extract much more unique data. The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extra…
1) I don't mind destroying the binding to get the best quality. Any idea how I do so?
2) I have a multipage double-sided scanner (fujitsu scansnap). would this be sufficient to do the scan portion?
3) Is there anything that determines the font of the book text and reproduces that somehow? and that deals with things like bold and italic and applies that either as markdown output or what have you?
4) how do you de-paginate the raw text to reflow into (say) an epub or pdf format that will paginate based on the output device (page size/layout) specification?
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#454You know what'd be fucking nice? The ability to turn Gemini off.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#455Shameless plug: I'm working on a startup in this space. But the bounding box problem hits close to home. We've found Unstructured's API gives pretty accurate box coordinates, and with some tweaks you can make them even better. The tricky part is implementing those tweaks without burning a hole in your wallet.
Re: Ingesting PDFs and why Gemini 2.0 changes everything
#456Earlier quoted context omitted.
This is "good enough" for Banks to use when doing due diligence. You'd be surprised how much noise is in the system with the current state of the art: algorithms/web scrapers and entire buildings of humans in places like India.
due diligence of this sort? https://en.wikipedia.org/wiki/Know_your_customer
KYC is an API you can pay for now. Works pretty well for the price, IIRC over 10k/month or something.