11–20 of 456 posts
--
[1]: https://github.com/google-gemini/cookbook/blob/a916686f95f43...
90% accuracy +/- 10%? What could that be useful for, that’s awfully low.
I wonder how this compares to open source models (which might be less accurate but even cheaper if self-hosted?), e.g. Llama 3.2. I'll see if I can run the benchmark. Also regarding the failure case in the footnote, I think Gemini actually got that right (or at least outperformed Reducto) - the original document seems to have what I call a "3D" table where the third axis is rows within each cell, and having multiple…
Maybe incremental processing of chunks of the table would have worked, with subsequent stitching, but if Gemini can just process it that would be pretty good.
My problem statement is:
- Injest PDFs, summarize, and extract important information.
- Have some way to overlay the extracted information on the pdf in the UI.
- User can provide feedback on the overlaid info by accepting or rejecting the highlights as useful or not.
- This info goes back in to the model for reinforced learning.
Hoping to find something that can make this more manageable.
Been toying with the flash model. Not the top model, but think it'll see plenty use due to the details. Wins on things other than top of benchmark logs * Generous free tier * Huge context window * Lite version feels basically instant However * Lite model seems more prone to repeating itself / looping * Very confusing naming e.g. {model}-latest worked for 1.5 but now its {model}-001? The lite has a date appended, the…
> * Huge context window But how well does it actually handle that context window? E.g. a lot of models support 200K context, but the LLM can only really work with ~80K or so of it before it starts to get confused.
Been toying with the flash model. Not the top model, but think it'll see plenty use due to the details. Wins on things other than top of benchmark logs * Generous free tier * Huge context window * Lite version feels basically instant However * Lite model seems more prone to repeating itself / looping * Very confusing naming e.g. {model}-latest worked for 1.5 but now its {model}-001? The lite has a date appended, the…
> * Huge context window But how well does it actually handle that context window? E.g. a lot of models support 200K context, but the LLM can only really work with ~80K or so of it before it starts to get confused.
Been toying with the flash model. Not the top model, but think it'll see plenty use due to the details. Wins on things other than top of benchmark logs * Generous free tier * Huge context window * Lite version feels basically instant However * Lite model seems more prone to repeating itself / looping * Very confusing naming e.g. {model}-latest worked for 1.5 but now its {model}-001? The lite has a date appended, the…
> * Huge context window But how well does it actually handle that context window? E.g. a lot of models support 200K context, but the LLM can only really work with ~80K or so of it before it starts to get confused.
This is what I have found as well. From what I've read, LLMS do not work well with images for specific details due to image encoders which are too lossy. (No idea if this is actually correct.) For now I guess you can use regular OCR to get bounding boxes.