Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SearXNG: A free internet metasearch engine

github.com

Post reply on HN
Print view View original

81–89 of 89 posts

Re: SearXNG: A free internet metasearch engine

#81

Post by jodoherty »

asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

Beautiful! Thank you for making this.

I've been trying to find something to use for enriching my own self-hosted LLMs and agentic tools with information I find useful. Metasearch tools like SearXNG make it less likely you'll get blocked by bot detection tools when finding information, but usually it's something I've already found, read, or seen that I want to incorporate into my tooling.

I came to the conclusion that a self-hosted content storage system with a search engine and a browser extension that can extract and save web page content and metadata was the ideal setup for me. Preferably with some sort of federated content sharing ability and the ability to import creative commons content like Wikipedia and Gutenberg.

This looks almost exactly like what I wanted.

It'll be a few weeks before I have time to audit the code and deploy it, but I'm really looking forward to trying it out.

Re: SearXNG: A free internet metasearch engine

#82

Post by baranul »

RandyOrion wrote: post #62

I've been using searxng for several years now. I don't run my own instances because the inhumane network censorship imposed by GFW, and proxy detection enforced by search engines. Instead, I rely on public instances on the list [1] and libredirect [2]. Note that service from a single instance is not guaranteed, but you can always switch to other available instances with little cost within a minute. I won't say searxn…

The thing about the public instances, is now you often have to go through a lot of them to verify they work properly. SearXNG needs better quality control.

Often have to go through the preferences to deselect search engines that don't work (often because of the instance being blocked) or select those that do work, because of reliability problems. Which engines are working, can be different for each public instance, so that even saving a preference hash doesn't always work.

Would be great if SearXNG did automatic adjustment of presented search engines (or offered the option) based on reliability.

Re: SearXNG: A free internet metasearch engine

#83

Post by Havoc »

I've got SearXNG set up but lately I've been using uruky instead.

For those not familiar think EU-ish Kagi...but crucially their plans comes with very legit API limits too. Their recent show HN: https://news.ycombinator.com/item?id=48396004

Not on the same wavelength as searxng stallman wise but I prefer it for AI use. Faster, better quality (subjective) and much less tokens per search - 3.5k vs 23k. Searxng returns a bunch of extra crap that just pollutes AI context. e.g. Link to the thumbnail of the page. I guess you can build a proxy to clean it up but meh.

[not affiliated]

Re: SearXNG: A free internet metasearch engine

#84

Post by satvikpendem »

asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

Looks good, but was also curious about the "limitations of the metasearch concept," could you explain this more?

Re: SearXNG: A free internet metasearch engine

#85

Post by satvikpendem »

goodroot wrote: post #24

This appears to be a key tool for providing search to local models. I'm curious what setups folks use to provide this functionality. Since the quantized 24B parameter Gemma model came out, I've had good luck with tool calling on a 4070 Ti Super. Successful tool calling is what finally made the local experience useful. I should note this is for the general and not coding specific context.

TinySearch MCP. If you use Unsloth Studio however it simply calls DuckDuckGo's HTML API instead and works pretty well.

Re: SearXNG: A free internet metasearch engine

#86

Post by satvikpendem »

drnick1 wrote: post #14
satvikpendem wrote: post #2

TinySearch wraps this and works well for agents. It's better than the native SearXNG MCP because it optimizes the context before it even gets to the agent so as to not waste tokens. https://github.com/MarcellM01/TinySearch

SearXNG did not include a built-in MCP server, last time I checked.

You're right, I was previously using "MCP-SearXNG" and must've thought it was official for some reason.

Re: SearXNG: A free internet metasearch engine

#87

Post by ydj »

phrotoma wrote: post #74
ydj wrote: post #54

Earlier quoted context omitted.

Hister sounds like something I wanted for a while, but never got around to building. Searching stuff I’ve seen before is most of what I do with a search engine, so having it local and fast would be amazing. Eager to give it a try.

And the number of times I've searched for something that I saw a while ago but is now gone is way too damned high.

Yes, if it does archiving too that would be perfect.

Re: SearXNG: A free internet metasearch engine

#88

Post by RandyOrion »

baranul wrote: post #82
RandyOrion wrote: post #62

I've been using searxng for several years now. I don't run my own instances because the inhumane network censorship imposed by GFW, and proxy detection enforced by search engines. Instead, I rely on public instances on the list [1] and libredirect [2]. Note that service from a single instance is not guaranteed, but you can always switch to other available instances with little cost within a minute. I won't say searxn…

The thing about the public instances, is now you often have to go through a lot of them to verify they work properly. SearXNG needs better quality control. Often have to go through the preferences to deselect search engines that don't work (often because of the instance being blocked) or select those that do work, because of reliability problems. Which engines are working, can be different for each public instance, s…

Yeah, searxng instance reliability problem goes deeper than simple uptime. The google response time function on public instance page [1], a good measure of the search engine availability of one instance, is broken for some time. Without real testing, people really cannot get the full view of the service status of a single instance.

[1] https://searx.space

Re: SearXNG: A free internet metasearch engine

#89

Post by renegat0x0 »

asciimoo wrote: post #25

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister ( https://hister.org/ ). Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews…

I also use my own, lets say search engine.

I download only title, description, thumbnail, common og fields.

My index is very lean, I think. I have 2m of pages crawled.

https://github.com/rumca-js/Internet-Places-Database

It has tags, and votes support.

Recently I also launched my first fdroid app

https://github.com/rumca-js/OfflineWebSearch

https://f-droid.org/en/packages/io.github.rumcajs.offlineweb...

Post reply on HN