Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

github.com

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#11

Post by Cheer2171 »

zmccormick7 wrote: post #3
skanga wrote: post #2

You mentioned that spRAG uses OpenAI for embeddings, Claude 3 Haiku for AutoContext, and Cohere for reranking. Can you explain why & how did you make those choices?

Those are just the defaults, and spRAG is designed to be flexible in terms of the models you can use with it. For AutoContext (which is just a summarization task) Haiku offers a great balance of price and performance. Llama 3-8B would also be a great choice there, especially if you want something you can run locally. For reranking, the Cohere v3 reranker is by far the best performer on the market right now. And for e…

I bet you'll get a lot more adoption if you put info about using it with local self-hosted LLMs there. I'll never trust a cloud service with the documents I want to RAG.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#12

Post by zmccormick7 »

serjester wrote: post #10

Does this mean you're winding down your business? Just curious what the motivation to open source this was given this seems like your guy's core value add? Congrats on the launch.

That's a great question. I'll start with a little context: most of the users of our existing hosted platform are no-code/low-code developers who choose us because we're the simplest solution for building what they want to build (primarily because we have end-to-end workflows like Chat built-in). The improved retrieval performance is a nice-to-have for this group, but usually not the primary reason they choose us.

For the larger companies and venture-backed startups we've talked to, they almost universally want to own their RAG stack in-house or build on open-source frameworks, rather than outsource it to an end-to-end solution provider. So open-sourcing our core retrieval tech is our bid to appeal to these developers.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#14

Post by jwuphysics »

How much do you expect auto-context and clustering+re-ranking to help for cases in which documents already have high-quality summaries? For context, I parse astrophysics research papers from arXiv and simply embed by paper abstracts (which must be of a certain size), and then append (parts of) the rest of the paper for RAG.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#15

Post by zmccormick7 »

jwuphysics wrote: post #14

How much do you expect auto-context and clustering+re-ranking to help for cases in which documents already have high-quality summaries? For context, I parse astrophysics research papers from arXiv and simply embed by paper abstracts (which must be of a certain size), and then append (parts of) the rest of the paper for RAG.

So the point of AutoContext is so you don't have to do that two-step process of first finding the right document, and then finding the right section of that document. I think it's cleaner to do it this way, but it's not necessarily going to perform any better or worse. But then spRAG also has the RSE part which is what identifies the right section(s) of the document. Whether or not that helps in your case is going to depend on how good of a solution you already have for that.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#16

Post by zmccormick7 »

Cheer2171 wrote: post #11
zmccormick7 wrote: post #3

Earlier quoted context omitted.

Those are just the defaults, and spRAG is designed to be flexible in terms of the models you can use with it. For AutoContext (which is just a summarization task) Haiku offers a great balance of price and performance. Llama 3-8B would also be a great choice there, especially if you want something you can run locally. For reranking, the Cohere v3 reranker is by far the best performer on the market right now. And for e…

I bet you'll get a lot more adoption if you put info about using it with local self-hosted LLMs there. I'll never trust a cloud service with the documents I want to RAG.

Agreed. I've gotten a lot of feedback along those lines today, so that's my top priority now.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#17

Post by jwuphysics »

zmccormick7 wrote: post #15
jwuphysics wrote: post #14

How much do you expect auto-context and clustering+re-ranking to help for cases in which documents already have high-quality summaries? For context, I parse astrophysics research papers from arXiv and simply embed by paper abstracts (which must be of a certain size), and then append (parts of) the rest of the paper for RAG.

So the point of AutoContext is so you don't have to do that two-step process of first finding the right document, and then finding the right section of that document. I think it's cleaner to do it this way, but it's not necessarily going to perform any better or worse. But then spRAG also has the RSE part which is what identifies the right section(s) of the document. Whether or not that helps in your case is going to…

That makes sense and I'll run a few evals. Many thanks for open sourcing your work!

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#18

Post by _akhe »

zmccormick7 wrote: post #8
esafak wrote: post #6

I'd replace the "challenging real-world tasks" in the title with "dense text, like financial reports and legal documents". It sounds less general but that's a good thing. The repo is only two weeks old, and looks it, so how do you think spRAG distinguishes itself? This is a crowded space with more established players. The "vanilla RAG" benchmark figure you cite is not convincing because it can not be verified. Please…

That's great feedback. I actually went back and forth between those two descriptions. I agree that "dense text, like financial reports and legal documents" is more precise. Those are the kinds of use cases this project is built for. I want to keep this project tightly scoped to just retrieval over dense unstructured text, rather than trying to build a fully-featured RAG framework.

FWIW nobody has created a great JavaScript framework experience yet. Closest we have for RAG is LlamaIndexTS or Langchainjs but both are full of bugs and have little LLM support. Their whole approach to supporting LLMs is writing bespoke wrappers for each.

Maybe it's the same on the Python side, but it feels like nobody has nailed the perfect LLM wrapper library yet. I would focus on dev experience - make it dead simple to load in files and use it from 0-1.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#19

Post by uptownfunk »

zmccormick7 wrote: post #12
serjester wrote: post #10

Does this mean you're winding down your business? Just curious what the motivation to open source this was given this seems like your guy's core value add? Congrats on the launch.

That's a great question. I'll start with a little context: most of the users of our existing hosted platform are no-code/low-code developers who choose us because we're the simplest solution for building what they want to build (primarily because we have end-to-end workflows like Chat built-in). The improved retrieval performance is a nice-to-have for this group, but usually not the primary reason they choose us. For…

Uh, isn’t there a huge venture backed startup that rhymes with spleen that contradicts this? Not saying you’re wrong but.. one of you is.
Post reply on HN