Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From zero to a RAG system: successes and failures

en.andros.dev

Post reply on HN
Print view View original

51–60 of 129 posts

Re: From zero to a RAG system: successes and failures

#51

Post by sthimons »

z02d wrote: post #4

Maybe a bit off-topic: For my PhD, I wanted to leverage LLMs and AI to speed up the literature review process*. Due to time constraints, this never really lifted off for me. At the time I checked (about 6 months ago), several tools were already available (NotebookLM, Anara, Connected Papers, ZotAI, Litmaps, Consensus, Research Rabbit) supporting Literature Review. They have all pros and cons (and different scopes), b…

Oh! Same! I made an R / Shiny powered RAG/ Researching app that hooks into OpenAlex (for papers) and allows you to generate NotebookLM like outputs. Just got slides with from-paper images to be injected in, super fun. Takes an OpenRouter or local LLMs (if that's your thing). Network graphs too! https://github.com/seanthimons/serapeum/

Re: From zero to a RAG system: successes and failures

#52

Post by smrtinsert »

stingraycharles wrote: post #40
smrtinsert wrote: post #37

What would it look like to regularly react to source data changes? Seems like a big missing piece. Event based? regular cadence? Curious what people choose. Great post though.

Depends on the use case, ie frequency and impact of changes. Typically you would have a reindex process, and you keep track of hashes of chunks to check if you’ve already calculated this exact block before to avoid extra costs. And then run such a reindex process pretty frequently as it’s cheap / costs nothing when there are no changes.

makes great sense, thanks!

Re: From zero to a RAG system: successes and failures

#53

Post by lukewarm707 »

civeng wrote: post #20

Great write-up. Thank you! I’m contemplating a similar RAG architecture for my engineering firm, but we’re dealing with roughly 20x the data volume (estimating around 9TB of project files, specs, and PDFs). I've been reading about Google's new STATIC framework (sparse matrix constrained decoding) and am really curious about the shift toward generative retrieval for massive speedups well beyond this approach. For thos…

9tb should be fine for vectordb, for sure. google search is many petabytes of index with vector+semantic search, that is using ScaNN.

you could probably use the hybrid search in llamaindex; or elasticsearch. there is an off the shelf discovery engine api on gcp. vertex rag engine is end to end for building your own. gcp is too expensive though. alibaba cloud have a similar solution.

Re: From zero to a RAG system: successes and failures

#55

Post by lukewarm707 »

z02d wrote: post #4

Maybe a bit off-topic: For my PhD, I wanted to leverage LLMs and AI to speed up the literature review process*. Due to time constraints, this never really lifted off for me. At the time I checked (about 6 months ago), several tools were already available (NotebookLM, Anara, Connected Papers, ZotAI, Litmaps, Consensus, Research Rabbit) supporting Literature Review. They have all pros and cons (and different scopes), b…

onyx is good for this, it is standard doc ingestion -> chunk -> embedding -> index -> query -> rerank -> answer.

there are a few other local apps with simple knowledge base type things you can use with pdfs. cherry studio is nice, no reranking though.

Re: From zero to a RAG system: successes and failures

#57

Post by axus »

Nihilartikel wrote: post #14
JKCalhoun wrote: post #10

And some have been saying that RAGs are obsolete—that the context window of a modern LLM is adequate (preferable?). The example I recently read was that the contexts are large enough for the entire "The Lord of the Rings" books. That may be, but then there's an entire law library, the entirety of Wikipedia (and the example in this article of 451 GB). Surely those are at least an order of magnitude larger than Tolkien…

I'm not super deep on LLM development, but with ram being a material bottleneck and from what I've read about DeepSeek's results with offloading factual knowledge with 'engrams' I think that the near future will start moving towards the dense core of LLMs focusing much more on a distillation of universal reasoning and logic while factual knowledge is pushed out into slower storage. IIRC Nvidia's Nemotron Cascade is t…

Were he still corporeal, L. Ron would be all over this AI stuff.

Re: From zero to a RAG system: successes and failures

#58

Post by ravikirany22 »

dgb23 wrote: post #43
JKCalhoun wrote: post #10

And some have been saying that RAGs are obsolete—that the context window of a modern LLM is adequate (preferable?). The example I recently read was that the contexts are large enough for the entire "The Lord of the Rings" books. That may be, but then there's an entire law library, the entirety of Wikipedia (and the example in this article of 451 GB). Surely those are at least an order of magnitude larger than Tolkien…

Also the thing with context is that you want to keep it focused on the task at hand. For example there's evidence that typical use of AGENTS.md actually doesn't improve outcomes but just slows the LLMs down and confuses them. In my personal testing and exploration I found that small (local) LLMs perform drastically better, both in accuracy and speed, with heavily pruned and focused context. Just because you can fill…

This matches what we've been seeing empirically. The issue isn't just quantity of context — it's staleness. AGENTS.md and CLAUDE.md that reference renamed functions, deleted interfaces, or outdated patterns actively mislead the model with confident but wrong information.We've been auditing TypeScript repos and finding 10-84% of symbol references in AI config files are stale. A model reading a CLAUDE.md that says "use UserService.createUser()" when that function was renamed three weeks ago isn't just getting irrelevant context — it's getting a confident lie.The quality problem is probably as significant as the quantity problem, maybe more so.

Re: From zero to a RAG system: successes and failures

#59

Post by ravikirany22 »

smrtinsert wrote: post #37

What would it look like to regularly react to source data changes? Seems like a big missing piece. Event based? regular cadence? Curious what people choose. Great post though.

For code specifically this is the hardest part — the "source data" (the codebase) changes constantly with every commit, but the AI config files that describe it don't update automatically.The approach that works best is AST-diffing rather than hash-based reindexing — you can detect semantic changes (function renamed, interface deleted) rather than just textual changes, which gives you much more precise invalidation signals.

Re: From zero to a RAG system: successes and failures

#60

Post by jgalt212 »

JKCalhoun wrote: post #10

And some have been saying that RAGs are obsolete—that the context window of a modern LLM is adequate (preferable?). The example I recently read was that the contexts are large enough for the entire "The Lord of the Rings" books. That may be, but then there's an entire law library, the entirety of Wikipedia (and the example in this article of 451 GB). Surely those are at least an order of magnitude larger than Tolkien…

> some have been saying that RAGs are obsolete

I suspect the people saying that have not been transparent with their incentives.

Post reply on HN