Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SearXNG in Rust

github.com

Post reply on HN
Print view View original

21–25 of 25 posts

Re: SearXNG in Rust

#21

Post by alethic »

snailmailman wrote: post #13

I’ve always been tempted to set up SearXNG, but all the search engines already make me do captchas and anti-bot checkboxes every few searches. Wouldn’t a meta search engine run into the same issues? Or if I had a local server scraping search results pages wouldn’t I instantly get banned from all the major searches? I guess I already feel pseudo-banned from them so maybe it’s not actually a problem. Just a few minutes…

No, they do a pretty good job at getting around it. I only run into issues with Brave/Startpage, Google/DuckDuckGo/Wikipedia all work fine. And when you're fusing together multiple sources getting CAPTCHA'd is not actually a big issue, it's more so a thing to fix later.

I would say also -- if search engines are making you do CAPTCHAs every few searches, this is probably an issue with your exit node. I notice a sizable increase in CAPTCHA barriers when I'm on my VPN (most annoyingly Google/YouTube, which sometimes will serve you an infinite number of CAPTCHAs w/o letting you through).

By contrast, my SearXNG instance is making ~100 searches a day from a known good IP address (being my VPS), so rate limiting is less of a problem. You can also set up API keys for many search engines.

Re: SearXNG in Rust

#22

Post by alethic »

Cool project! The title is not quite accurate though, the project is "a metadata search engine similar to searxng in rust". It doesn't look like it implements / intends to implement any of SearXNG's advanced features, like weighting of individual sites in results, wikidata integration, or... a frontend :-)

Re: SearXNG in Rust

#23

Post by snailmailman »

alethic wrote: post #21
snailmailman wrote: post #13

I’ve always been tempted to set up SearXNG, but all the search engines already make me do captchas and anti-bot checkboxes every few searches. Wouldn’t a meta search engine run into the same issues? Or if I had a local server scraping search results pages wouldn’t I instantly get banned from all the major searches? I guess I already feel pseudo-banned from them so maybe it’s not actually a problem. Just a few minutes…

No, they do a pretty good job at getting around it. I only run into issues with Brave/Startpage, Google/DuckDuckGo/Wikipedia all work fine. And when you're fusing together multiple sources getting CAPTCHA'd is not actually a big issue, it's more so a thing to fix later. I would say also -- if search engines are making you do CAPTCHAs every few searches, this is probably an issue with your exit node. I notice a sizabl…

I wish it was a problem with my vpn/exit node. But I get these captchas when I’m not on a VPN. I assume it’s just because I don’t login to Google and I enable tracking prevention stuff in Firefox? But that’s really only on my desktop. I get them on my iphone too which is vanilla iOS safari. I get captchas and the cloudflare checkboxes extremely frequently, and sometimes cannot get past them at all.

I saw the articles about residential proxies on TVs recently and honestly have been worried something like that was the cause, but I’ve kept an eye on my network traffic and I don’t notice anything suspicious. Pretty sure I’m not part of a botnet or proxy or anything; but maybe other nearby IPs are.

Maybe I’ll give SearxNG a try. Searching is bad enough as it is, it can’t make it too much worse.

Re: SearXNG in Rust

#24

Post by knowaveragejoe »

asciimoo wrote: post #4

Hi everyone, I am the original author of Searx. I'm working on a new search project, Hister with a similar motivation: reducing our dependence on external search engines while keeping searches and personal data under our control. Searx is a metasearch engine that forwards queries to other search providers. Hister takes a different approach. It builds a private full text index from content you choose, then searches th…

Is there any plan to incorporate the same kind of searching functionality that searx has? i.e. fan out to different engines/indexes?
Post reply on HN