Local Deep Research – ArXiv, wiki and other searches included
1–10 of 35 posts
Re: Local Deep Research – ArXiv, wiki and other searches included
#2Example output: https://github.com/LearningCircuit/local-deep-research/blob/...
Re: Local Deep Research – ArXiv, wiki and other searches included
#3Example output: https://github.com/LearningCircuit/local-deep-research/blob/...
You could be the first if you were to develop an eval (preferably automated with llm as judge) and compared local deep research with perplexity's, openai's and deepseek's implementations on high-information questions.
Re: Local Deep Research – ArXiv, wiki and other searches included
#4Example output: https://github.com/LearningCircuit/local-deep-research/blob/...
You could be the first if you were to develop an eval (preferably automated with llm as judge) and compared local deep research with perplexity's, openai's and deepseek's implementations on high-information questions.
How do they evaluate the quality of the report? It's one of the most important things for me.
Re: Local Deep Research – ArXiv, wiki and other searches included
#5I applaud the effort for the local (lo-fi) space ! Yet, reading over the example linked in the docs (which does not seem cheery-picked, kudos for that!), my impression is that the document is a rather messy outcome [1].
I think what's missing is one (or more) step in-between, possible a graph database (eg[2]), which the LLM can place all it's information in, see relevant interconnections, query to question itself, and then generate the final report.
(maybe the final report could be an interactive HTML file that the user can ask questions, or edit themselves).
There's also a similar open-deep research tool called onyx [2], with I think has better UI/UX albeit not local. Maybe the author could consider porting this to local instead of rolling and maintaining another deep-research tool themselves ?
I'm saying this, not because I think it's not a good project, but because there are a ton of open deep-research projects which I'm afraid will just fizzle out, and would be better if people would join forces working on those aspects they care most about (e.g. local aspect, or RAG strategies, etc) .
[1] https://github.com/LearningCircuit/local-deep-research/blob/...
[2] "In-Browser Graph RAG with Kuzu-WASM and WebLLM" https://news.ycombinator.com/item?id=43321523
Re: Local Deep Research – ArXiv, wiki and other searches included
#6Is anyone using (local) LLMs to directly search for (by scanning over) relevant materials from a corpus rather than relying on vector search?
Re: Local Deep Research – ArXiv, wiki and other searches included
#7Is anyone using (local) LLMs to directly search for (by scanning over) relevant materials from a corpus rather than relying on vector search?
Generally this fails.
Most LLMs lose the ability to track facts over about 20k words of content, the best can manage maybe 40k words.
Look for "needle" benchmark tests, as in needle-in-haystack.
Not to mention the memory requirements of such a huge context like 128k or 1M tokens. Only people with enterprise servers at home could run that locally.
Re: Local Deep Research – ArXiv, wiki and other searches included
#8Is anyone using (local) LLMs to directly search for (by scanning over) relevant materials from a corpus rather than relying on vector search?
Generally this fails. Most LLMs lose the ability to track facts over about 20k words of content, the best can manage maybe 40k words. Look for "needle" benchmark tests, as in needle-in-haystack. Not to mention the memory requirements of such a huge context like 128k or 1M tokens. Only people with enterprise servers at home could run that locally.
Very good answer. It is very hard with small LLM.
Re: Local Deep Research – ArXiv, wiki and other searches included
#9Nice work!
I’ve been thinking recently that a local collection of pre-processed for RAG using curated focused structured information might be a good complement to this dynamic searching approach.
I see this used LangChain, might be worth checking into txtai.
Re: Local Deep Research – ArXiv, wiki and other searches included
#10Earlier quoted context omitted.
You could be the first if you were to develop an eval (preferably automated with llm as judge) and compared local deep research with perplexity's, openai's and deepseek's implementations on high-information questions.
How do they evaluate the quality of the report? It's one of the most important things for me.
Given a benchmark corpus, the evaluation criteria could be:
- Facts extracted: the amount of relevant facts extracted from the corpus
- Interpretations : based on the facts, % of correct interpretations made
- Correct Predictions: based on the above, % of correct extrapolations / interpolations / predictions made
The ground truth could be in JSON file per example. (If the solution you want to benchmark uses a graph db, you could compare these aspects with a LLM as judge.)
---
The actual writing is more about formal/business/academic style, and I find less relevant for a benchmark.
However I would find it crucial to run a "reverse RAG" over the generated report to ensure each claim has a source. [0]
[0] https://venturebeat.com/ai/mayo-clinic-secret-weapon-against...