Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SearXNG: A free internet metasearch engine

github.com

Post reply on HN
Print view View original

41–50 of 89 posts

Re: SearXNG: A free internet metasearch engine

#41

Post by asciimoo »

operatingthetan wrote: post #26
asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

I installed this a while back and honestly I almost never touch it. It turns out that for me searching my history doesn't really replace a search engine at all. The built in extractor list is pretty limited and adding them seems like too much of an ordeal for me to bother.

Sure, it cannot fully replace web search engines (yet), but it can reduce the dependence on these services more and more as your index grows. Hister is designed to support quickly falling back to traditional search engines with a single hotkey if no results found.

I agree, we should add more extractors [1]. Can you recommend extractors you missed?

[1] https://github.com/asciimoo/hister/issues/305

Re: SearXNG: A free internet metasearch engine

#42

Post by asciimoo »

justusthane wrote: post #37
zeroq wrote: post #36

Earlier quoted context omitted.

I'm sorry for not taking the time to read the docs, but I have a question. Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name. For years I've…

Also very interested in this. I was playing around with doing the same thing with YaCY. I want the proxy aspect so that I can proxy my phone traffic through it as well.

Unfortunately mobile Chrome browsers don't support browser extensions, but our extension works well on mobile Firefox.

Re: SearXNG: A free internet metasearch engine

#43

Post by MrDrMcCoy »

asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

Always excited to see new things like Hister in the search space. What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful? Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?

Re: SearXNG: A free internet metasearch engine

#44

Post by asciimoo »

MrDrMcCoy wrote: post #43
asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

Always excited to see new things like Hister in the search space. What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful? Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?

> What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful?

There has been no stress tests in this regard. The indexer lib Bleve [1] can handle millions of documents according to their documentation.

> Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?

My long term goal is exactly this. I'd like to add federation/P2P feature [2][3] to evolve from being a private search companion. I'd appreciate any help designing the system.

[1] https://blevesearch.com/docs/Home/ [2] https://github.com/asciimoo/hister/discussions/432 [3] https://hister.org/posts/public-search

Re: SearXNG: A free internet metasearch engine

#45

Post by zeroq »

asciimoo wrote: post #39
zeroq wrote: post #36

Earlier quoted context omitted.

I'm sorry for not taking the time to read the docs, but I have a question. Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name. For years I've…

Absolutely, this is a great example where Hister can shine. I started Hister as a proxy as well, but quickly switched to the current extension based approach, because intercepting HTTPS traffic requires a MiTM proxy which is much more painful to setup than installing a browser extension.

would it be possible to gdrive/rsync/git the data between machines and then use the data on an online server for retrieval (given that I would handle data sync myself)?

also what exactly are you using for search? does it support trigrams? how do you sort results?

Re: SearXNG: A free internet metasearch engine

#46

Post by derrida »

asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

Wow! that looks like a bit of software I have been dreaming about for awhile - will definately check out! You're at least doing something right in communicating the reasons why and appeal for starters! All the best!

Re: SearXNG: A free internet metasearch engine

#47

Post by blackqueeriroh »

asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

Oh thank god there used to be several tools like this and they slowly went away. I’ve been wanting this to return.

Re: SearXNG: A free internet metasearch engine

#48

Post by Havoc »

drnick1 wrote: post #14
satvikpendem wrote: post #2

TinySearch wraps this and works well for agents. It's better than the native SearXNG MCP because it optimizes the context before it even gets to the agent so as to not waste tokens. https://github.com/MarcellM01/TinySearch

SearXNG did not include a built-in MCP server, last time I checked.

It does have json response though so pretty trivial to get an LLM to make you a mcp

Re: SearXNG: A free internet metasearch engine

#49

Post by jared0x90 »

goodroot wrote: post #24

This appears to be a key tool for providing search to local models. I'm curious what setups folks use to provide this functionality. Since the quantized 24B parameter Gemma model came out, I've had good luck with tool calling on a 4070 Ti Super. Successful tool calling is what finally made the local experience useful. I should note this is for the general and not coding specific context.

are you running a quant?

i have a friend with a 4080 that is wanting to experiment with local models and those cards should be similar enough. can you give any more detail about your setup? ty!

Re: SearXNG: A free internet metasearch engine

#50

Post by c-hendricks »

drnick1 wrote: post #31
goodroot wrote: post #24

This appears to be a key tool for providing search to local models. I'm curious what setups folks use to provide this functionality. Since the quantized 24B parameter Gemma model came out, I've had good luck with tool calling on a 4070 Ti Super. Successful tool calling is what finally made the local experience useful. I should note this is for the general and not coding specific context.

I am also interested in what a full local AI stack with web search and other tools looks like. As far as I can tell, SearX does not embed an MCP server, so it can't be directly called from llama-server for example. Open WebUI does have an integration for SearX and other providers, but the results I obtained weren't particularly impressive.

I use Searxng through Onyx, both as regular search and Onyx's Deep Research mode. I also have https://github.com/ihor-sokoliuk/mcp-searxng to add search to coding agents. Haven't really had many issues with it.
Post reply on HN