is this project like yacy? i mean run your own crawler to build your own search engine? nice work
Show HN: Wiby is now free software
21–30 of 36 posts
Re: Show HN: Wiby is now free software
#22the guide linked in the Readme is comprehensive and a good read - also gives instructions on who to scale and distribute load. If the author can answer: how big did the fulltext table became for x entries on wiby.me - and what is a common response time on N amount of searches per minute for this dataset? Would you offer a /traffic or /stats page within about/ ? duckduckgo shows traffic, not index stats though. I don'…
>how big did the fulltext table became for x entries on wiby.me I want Wiby to be comprised mainly of human submitted pages, so for 99% of the index, only the pages submitted by users are indexed and no further crawling was done. However I recognized that not having the capability to crawl through links would not make it useful for others, so I added in the crawling capability to my liking and tested it accordingly.…
What is the botnet owner thinking of gaining from a small potatoes search engine? Seems rather futile?
Re: Show HN: Wiby is now free software
#23Earlier quoted context omitted.
>how big did the fulltext table became for x entries on wiby.me I want Wiby to be comprised mainly of human submitted pages, so for 99% of the index, only the pages submitted by users are indexed and no further crawling was done. However I recognized that not having the capability to crawl through links would not make it useful for others, so I added in the crawling capability to my liking and tested it accordingly.…
>Wiby is being barraged by a nasty spam botnet with thousands of constantly changing IPs. Short of having a private beta like kagi, how else could those bot nets be excluded? How difficult is it to create a white list of uninfected IPs?
For what it's worth, I ended up putting Marginalia Search behind Cloudflare to deal with what I assume is the same group. At worst I saw 30k queries per hour.
Re: Show HN: Wiby is now free software
#24This is really amazing, and can allow for niche communities to build up "useful links" that can be easily searched, WITHOUT having to use outside search engines.
>There are several forms to control the search engine. There is no central form linking everything together, just a collection of different folders that you can rename if you want.
Maybe a good addition would be /admin/ that links all of the admin functions.
Re: Show HN: Wiby is now free software
#25Earlier quoted context omitted.
>Wiby is being barraged by a nasty spam botnet with thousands of constantly changing IPs. Short of having a private beta like kagi, how else could those bot nets be excluded? How difficult is it to create a white list of uninfected IPs?
For what it's worth, I ended up putting Marginalia Search behind Cloudflare to deal with what I assume is the same group. At worst I saw 30k queries per hour.
Who the hell has the incentive to shut down by force small, independent search engines? Competitors?
Re: Show HN: Wiby is now free software
#26Awesome, thank you for publishing it!
The internet, somewhat ironically, really needs a search engine that works in the current day. You can't find anything anymore. It's like Google has been un-invented.
Hopefully some day soon the internet will be searchable again.
Re: Show HN: Wiby is now free software
#27Earlier quoted context omitted.
For what it's worth, I ended up putting Marginalia Search behind Cloudflare to deal with what I assume is the same group. At worst I saw 30k queries per hour.
Who the hell has the incentive to shut down by force small, independent search engines? Competitors?
My unsubstantiated hunch based on looking at the types of queries, which at least for me were over-specified as all hell and within the sphere of pharmaceuticals, e-shopping and the like, is that they're gambling on the search engine being backed by Google or Bing, and they're effectively trying to poison their typeahead suggestion data.
I'd guess they're just aiming their gatling gun at whatever sites has an opensearch specification without much oversight.
It's also crossed my mind it might be some sketchy law firm looking for DMCA violations, since a fair bunch of the queries looked like they were after various forms of contraband. Seems weird they'd use a botnet though. Like most of the IPs seemed to be like enterprise routers with public facing admin pages and the like. Does not seem above board at all.
Re: Show HN: Wiby is now free software
#28Awesome, thank you for publishing it! The internet, somewhat ironically, really needs a search engine that works in the current day. You can't find anything anymore. It's like Google has been un-invented. Hopefully some day soon the internet will be searchable again.
Google works fine for me and seemingly most of the population.
Re: Show HN: Wiby is now free software
#29Awesome, thank you for publishing it! The internet, somewhat ironically, really needs a search engine that works in the current day. You can't find anything anymore. It's like Google has been un-invented. Hopefully some day soon the internet will be searchable again.
Google works fine for me and seemingly most of the population.
I'll second the original comment, for me Google is trash that won't find anything except blogspam and mainstream sites.
Read a thing on a blog once? Good luck, you're never finding that again.
Re: Show HN: Wiby is now free software
#30This looks really great, nice work. Wish it could be installed on basic shared hosting though.