Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-Source DocumentAI with Ollama

rlama.dev

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Show HN: Open-Source DocumentAI with Ollama

#31

Post by simonw »

andai wrote: post #28

This appears to do no chunking. It just shoves the entire document (entire book, in my case) into the embedding request to Ollama. So it's only helpful if all your documents are small (i.e. no books). The embedding model (bge-m3 in this case) has a sequence length of 8192 tokens, i.e. rlama tries to embed the whole book, but Ollama can only put the first few pages into the embedding request. Then when retrieving, it…

Sadly, the hardest part of running local models with tools like Ollama appears to be longer context prompts.

Models that respond really quickly to a short sentence prompt need vastly more RAM and CPU/GPU time for significantly longer inputs. I'm finding this really damages their utility for me.

Re: Show HN: Open-Source DocumentAI with Ollama

#32

Post by elliot07 »

andai wrote: post #28

This appears to do no chunking. It just shoves the entire document (entire book, in my case) into the embedding request to Ollama. So it's only helpful if all your documents are small (i.e. no books). The embedding model (bge-m3 in this case) has a sequence length of 8192 tokens, i.e. rlama tries to embed the whole book, but Ollama can only put the first few pages into the embedding request. Then when retrieving, it…

What's the gold standard paid offering that does this?

Re: Show HN: Open-Source DocumentAI with Ollama

#33

Post by danihh »

elliot07 wrote: post #32
andai wrote: post #28

This appears to do no chunking. It just shoves the entire document (entire book, in my case) into the embedding request to Ollama. So it's only helpful if all your documents are small (i.e. no books). The embedding model (bge-m3 in this case) has a sequence length of 8192 tokens, i.e. rlama tries to embed the whole book, but Ollama can only put the first few pages into the embedding request. Then when retrieving, it…

What's the gold standard paid offering that does this?

Not a paid solution, but great for testing models yourself: AWS bedrock.

Wonky documentation (definitely released too early), but imo the best model agnostic diy solution out there.

Re: Show HN: Open-Source DocumentAI with Ollama

#34

Post by jwithington »

Say that I'm an amateur historian. I go into an archive and scan a bunch of documents (letters, diagrams, maps, etc). They're saved as JPG files.

What's the best way to make sense of that corpus of knowledge? Is it Rlama or something else?

I'm currently trying to homebrew this with Gemini, but am not sure if there's something that gets me out of building a RAG system from scratch.

Re: Show HN: Open-Source DocumentAI with Ollama

#36

Post by codelion »

andai wrote: post #28

This appears to do no chunking. It just shoves the entire document (entire book, in my case) into the embedding request to Ollama. So it's only helpful if all your documents are small (i.e. no books). The embedding model (bge-m3 in this case) has a sequence length of 8192 tokens, i.e. rlama tries to embed the whole book, but Ollama can only put the first few pages into the embedding request. Then when retrieving, it…

yeah, chunking seems to be the key for any decent RAG implementation... it's interesting how much the retrieval strategy impacts the final answer quality. i've seen some community members mention that even with chunking, things like chunk overlap and smart metadata can significantly improve results. also, presenting search results to the user alongside the AI summary is a great point.

Re: Show HN: Open-Source DocumentAI with Ollama

#37

Post by Dontizi »

codelion wrote: post #36
andai wrote: post #28

This appears to do no chunking. It just shoves the entire document (entire book, in my case) into the embedding request to Ollama. So it's only helpful if all your documents are small (i.e. no books). The embedding model (bge-m3 in this case) has a sequence length of 8192 tokens, i.e. rlama tries to embed the whole book, but Ollama can only put the first few pages into the embedding request. Then when retrieving, it…

yeah, chunking seems to be the key for any decent RAG implementation... it's interesting how much the retrieval strategy impacts the final answer quality. i've seen some community members mention that even with chunking, things like chunk overlap and smart metadata can significantly improve results. also, presenting search results to the user alongside the AI summary is a great point.

This is my next step. Currently, I’ve built an MVP to test the features, integrations, and see how far I can go with rLlama. I’m already developing a RAG on my end by chunking the data, adding overlap, and using metadata to retrieve the best possible context. This should be deployed soon. The version on GitHub has been pushed for days now, and it was only a version to showcase the features. I can’t wait to improve it and make it useful for everyone!

Re: Show HN: Open-Source DocumentAI with Ollama

#38

Post by Dontizi »

andai wrote: post #28

This appears to do no chunking. It just shoves the entire document (entire book, in my case) into the embedding request to Ollama. So it's only helpful if all your documents are small (i.e. no books). The embedding model (bge-m3 in this case) has a sequence length of 8192 tokens, i.e. rlama tries to embed the whole book, but Ollama can only put the first few pages into the embedding request. Then when retrieving, it…

I have just implemented chunking with overlap for larger documents to split texts into smaller chunks and ensure access to all documentation in your RAG. It's currently in the testing phase, and I’d like to experiment with different models to optimize the process. Once I confirm that everything is working correctly, I can merge the PR into the main branch, and you’ll just need to update Rlama with `rlama update`.

Re: Show HN: Open-Source DocumentAI with Ollama

#39

Post by [deleted] »

Dontizi wrote: post #38
andai wrote: post #28

This appears to do no chunking. It just shoves the entire document (entire book, in my case) into the embedding request to Ollama. So it's only helpful if all your documents are small (i.e. no books). The embedding model (bge-m3 in this case) has a sequence length of 8192 tokens, i.e. rlama tries to embed the whole book, but Ollama can only put the first few pages into the embedding request. Then when retrieving, it…

I have just implemented chunking with overlap for larger documents to split texts into smaller chunks and ensure access to all documentation in your RAG. It's currently in the testing phase, and I’d like to experiment with different models to optimize the process. Once I confirm that everything is working correctly, I can merge the PR into the main branch, and you’ll just need to update Rlama with `rlama update`.

[deleted]
Post reply on HN