Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SearXNG: A free internet metasearch engine

github.com

Post reply on HN
Print view View original

31–40 of 89 posts

Re: SearXNG: A free internet metasearch engine

#31

Post by drnick1 »

goodroot wrote: post #24

This appears to be a key tool for providing search to local models. I'm curious what setups folks use to provide this functionality. Since the quantized 24B parameter Gemma model came out, I've had good luck with tool calling on a 4070 Ti Super. Successful tool calling is what finally made the local experience useful. I should note this is for the general and not coding specific context.

I am also interested in what a full local AI stack with web search and other tools looks like. As far as I can tell, SearX does not embed an MCP server, so it can't be directly called from llama-server for example. Open WebUI does have an integration for SearX and other providers, but the results I obtained weren't particularly impressive.

Re: SearXNG: A free internet metasearch engine

#32

Post by asciimoo »

kristianpaul wrote: post #29
asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

Is this similar to fastcrw ?

Both are search engines, but that's all the similarity. Hister has a traditional crawler, but its biggest strength is automatically indexing browser tabs as those are rendered. This way it bypasses authentication, CloudFlare, captchas and most of the annoying limitations of traditional crawlers. Hister also provides full offline result previews. Check out the small read-only demo: https://demo.hister.org/

Re: SearXNG: A free internet metasearch engine

#33

Post by gardnr »

goodroot wrote: post #24

This appears to be a key tool for providing search to local models. I'm curious what setups folks use to provide this functionality. Since the quantized 24B parameter Gemma model came out, I've had good luck with tool calling on a 4070 Ti Super. Successful tool calling is what finally made the local experience useful. I should note this is for the general and not coding specific context.

It has a JSON mode that you need to enable in settings and then you can create a simple python script to interact with it or have the agent use `curl` and `jq` to interact with it.

It's at the bottom of this page: https://docs.searxng.org/admin/settings/settings_search.html

Re: SearXNG: A free internet metasearch engine

#35

Post by chrisss395 »

asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

I love your idea and wondered why saving and indexing browser visited pages was not being done. Does this handle large amounts of local files, for example 10-20TB across file types like Powerpoint, Excel, Word, and PDF?

Re: SearXNG: A free internet metasearch engine

#36

Post by zeroq »

asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

I'm sorry for not taking the time to read the docs, but I have a question.

Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name.

For years I've been collecting links to different work related trivia which I use on a daily basis as a rabbit-from-a-hat solution to answer random question from friends and coworkers. For example someone randomly asked me for an idea for color palette for data charts and I can immediately give them a scientific research into the color palette. Or an obscure algorithm.

But with time the collection has grown substantially and it's really cumbersome to find the proper things.

Would your project be a good fit for my problem?

Re: SearXNG: A free internet metasearch engine

#37

Post by justusthane »

zeroq wrote: post #36
asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

I'm sorry for not taking the time to read the docs, but I have a question. Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name. For years I've…

Also very interested in this. I was playing around with doing the same thing with YaCY. I want the proxy aspect so that I can proxy my phone traffic through it as well.

Re: SearXNG: A free internet metasearch engine

#38

Post by asciimoo »

chrisss395 wrote: post #35
asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

I love your idea and wondered why saving and indexing browser visited pages was not being done. Does this handle large amounts of local files, for example 10-20TB across file types like Powerpoint, Excel, Word, and PDF?

In its current form it cannot handle this amount of data efficiently (and doesn't support powerpoint/excel/word yet), but this is a valid use-case, I've added a TODO item to experiment with it.

Re: SearXNG: A free internet metasearch engine

#39

Post by asciimoo »

zeroq wrote: post #36
asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

I'm sorry for not taking the time to read the docs, but I have a question. Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name. For years I've…

Absolutely, this is a great example where Hister can shine.

I started Hister as a proxy as well, but quickly switched to the current extension based approach, because intercepting HTTPS traffic requires a MiTM proxy which is much more painful to setup than installing a browser extension.

Post reply on HN