Amazing！ Search is not a problem . What to search is! Using reasoning model, it is much easier to split task and focus on what to search
+1. Asking the right questions is always the most difficult thing to do.
DeepSearcher: A local open-source Deep Research
21–28 of 28 posts
Re: DeepSearcher: A local open-source Deep Research
#22I think the magic of Grok's implementation of this is that they already have most of the websites cached (guessing via their twitter crawler) so it all feels very snappy. Bing/Brave search don't seem to offer that in their search apis. Does such a thing exist as a service?
Web search APIs can't present the full document due to copyright. They can only present the snippet contextual to the query. I wrote my own implementation using various web search APIs and a puppeteer service to download individual documents as needed. It wasn't that hard but I do get blocked by some sites (reddit for example).
Google and Bing's Cache, Archive.org, Archive.is, CommonCrawl... many services have previously or currently presented the full document.
Google and Bing removed their cache features when LLMs started taking off – as I said in a sibling comment, I wonder if they felt that that regime was finally going to be challenged in court as people tried to protect their data.
That being said, "can't present the full document due to copyright" seems at odds with all of the above examples existing for years.
Re: DeepSearcher: A local open-source Deep Research
#23I think the magic of Grok's implementation of this is that they already have most of the websites cached (guessing via their twitter crawler) so it all feels very snappy. Bing/Brave search don't seem to offer that in their search apis. Does such a thing exist as a service?
exa is your answer i think? https://latent.space/p/exa
Re: DeepSearcher: A local open-source Deep Research
#24I think the magic of Grok's implementation of this is that they already have most of the websites cached (guessing via their twitter crawler) so it all feels very snappy. Bing/Brave search don't seem to offer that in their search apis. Does such a thing exist as a service?
exa is your answer i think? https://latent.space/p/exa
Re: DeepSearcher: A local open-source Deep Research
#25The real magic bullet would be searching lib-gen and sci-hub as well
Re: DeepSearcher: A local open-source Deep Research
#26I'm wondering about the practical implications of integrating web crawling. Could this, in theory, be used solely for reading papers from Sci-Hub and producing valid graduate-level research?
It could be useful for comparing reports built using DeepSeek R1 vs. GPT-4o and other large models. The code being open source might highlight the limitations of different LLMs much faster and help develop better reasoning loops in future prompts for specific needs. Really interesting stuff.
Re: DeepSearcher: A local open-source Deep Research
#27I think the magic of Grok's implementation of this is that they already have most of the websites cached (guessing via their twitter crawler) so it all feels very snappy. Bing/Brave search don't seem to offer that in their search apis. Does such a thing exist as a service?
Web search APIs can't present the full document due to copyright. They can only present the snippet contextual to the query. I wrote my own implementation using various web search APIs and a puppeteer service to download individual documents as needed. It wasn't that hard but I do get blocked by some sites (reddit for example).
(founder here) We are working on that problem of providing deeper level of search especially on proprietary datasets (think reference works, books, papers etc.). Started off with Arxive papers )We are working on that problem of providing deeper level of search especially on proprietary/ copyright datasets (think reference works, books, papers etc.). We are working with a number of large publishers on this.
We started off with Arxive papers to test out the product- would love to get feedback :)
Re: DeepSearcher: A local open-source Deep Research
#28This doesn't seem to use local LLMs... so it's not really local. :-\ Is there a deep searcher that can also use local LLMs like those hosted by Ollama and LM Studio?
[dead]