Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

github.com

Post reply on HN
Print view View original

1–10 of 24 posts

Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#1

Post by zmccormick7 »

Hey HN, I’m Zach from Superpowered AI (YC S22). We’ve been working in the RAG space for a little over a year now, and we’ve recently decided to open-source all of our core retrieval tech.

spRAG is a retrieval system that’s designed to handle complex real-world queries over dense text, like legal documents and financial reports. As far as we know, it produces the most accurate and reliable results of any RAG system for these kinds of tasks. For example, on FinanceBench, which is an especially challenging open-book financial question answering benchmark, spRAG gets 83% of questions correct, compared to 19% for the vanilla RAG baseline (which uses Chroma + OpenAI Ada embeddings + LangChain).

You can find more info about how it works and how to use it in the project’s README. We’re also very open to contributions. We especially need contributions around integrations (i.e. adding support for more vector DBs, embedding models, etc.) and around evaluation.

Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks
github.com

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#3

Post by zmccormick7 »

skanga wrote: post #2

You mentioned that spRAG uses OpenAI for embeddings, Claude 3 Haiku for AutoContext, and Cohere for reranking. Can you explain why & how did you make those choices?

Those are just the defaults, and spRAG is designed to be flexible in terms of the models you can use with it. For AutoContext (which is just a summarization task) Haiku offers a great balance of price and performance. Llama 3-8B would also be a great choice there, especially if you want something you can run locally. For reranking, the Cohere v3 reranker is by far the best performer on the market right now. And for embeddings, it's really a toss-up between OpenAI, Cohere, and Voyage.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#4

Post by cyanydeez »

Im planning a RAG and this seems to implement the "autocontext" i was expectong to do.

For the library i have, its not just the file name but multiple folder names are descriptive, especially if i build a data dictionary.

Have you looked into some simple tagging/conversion dictionary that preproccess the context?

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#5

Post by bitshaker »

Amazing. I’m looking at building an app to look over the employment sections of the legal code and come back with results if things are allowed or not.

Answering questions like:

Can my employer do X? As an employee in this country, what’s my minimum days off I can take?

And so on.

If your claims are true, then this will be exactly what I’m looking for.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#6

Post by esafak »

I'd replace the "challenging real-world tasks" in the title with "dense text, like financial reports and legal documents". It sounds less general but that's a good thing.

The repo is only two weeks old, and looks it, so how do you think spRAG distinguishes itself? This is a crowded space with more established players.

The "vanilla RAG" benchmark figure you cite is not convincing because it can not be verified. Please share your benchmarking code.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#7

Post by zmccormick7 »

cyanydeez wrote: post #4

Im planning a RAG and this seems to implement the "autocontext" i was expectong to do. For the library i have, its not just the file name but multiple folder names are descriptive, especially if i build a data dictionary. Have you looked into some simple tagging/conversion dictionary that preproccess the context?

In our AutoContext implementation, the document title gets included with the generated summary. So if you have files that are organized into nested folders with descriptive names, you can input that full file path as the `document_title`. I did this with one of our internal benchmarks and it worked really well.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#8

Post by zmccormick7 »

esafak wrote: post #6

I'd replace the "challenging real-world tasks" in the title with "dense text, like financial reports and legal documents". It sounds less general but that's a good thing. The repo is only two weeks old, and looks it, so how do you think spRAG distinguishes itself? This is a crowded space with more established players. The "vanilla RAG" benchmark figure you cite is not convincing because it can not be verified. Please…

That's great feedback. I actually went back and forth between those two descriptions. I agree that "dense text, like financial reports and legal documents" is more precise. Those are the kinds of use cases this project is built for.

I want to keep this project tightly scoped to just retrieval over dense unstructured text, rather than trying to build a fully-featured RAG framework.

Re: Show HN: SpRAG – Open-source RAG implementation for challenging real-world tasks

#9

Post by zmccormick7 »

bitshaker wrote: post #5

Amazing. I’m looking at building an app to look over the employment sections of the legal code and come back with results if things are allowed or not. Answering questions like: Can my employer do X? As an employee in this country, what’s my minimum days off I can take? And so on. If your claims are true, then this will be exactly what I’m looking for.

I think spRAG should be pretty well suited for that use case. I think the biggest challenge will be generating specific search queries off of more general user inputs. You can look at the `auto_query.py` file for a basic implementation of that kind of system, but it'll likely require some experimentation and customization for your use case.
Post reply on HN