11–20 of 36 posts
What is available to share crawls? There would be a lot of trust required to use the data for anything but things like Common Crawl save a lot of time. Does Wiby support starting with that?
Thank you so much for making Wiby! I have truly enjoyed what its brought!!
Wiby is amazing and one of my favorite ways to discover the more unique side of the internet. I love to kill time by clicking "Surprise me" and half the RSS feeds I subscribe to are from this! Thank you so much for making Wiby! I have truly enjoyed what its brought!!
Wow! Thank you so much for this, I will for sure try and play around with this. Is it possible to run the Crawler behind like socks5 proxy or such?
This is awesome - thanks for sharing. I just so happen to have a domain left lying around for the Searx instance I never spun up, so I might end up using it for a weekend project with this. Love Wiby, been playing with it awhile now. I especially enjoy the "surprise me" button from time to time.
the guide linked in the Readme is comprehensive and a good read - also gives instructions on who to scale and distribute load. If the author can answer: how big did the fulltext table became for x entries on wiby.me - and what is a common response time on N amount of searches per minute for this dataset? Would you offer a /traffic or /stats page within about/ ? duckduckgo shows traffic, not index stats though. I don'…
>how big did the fulltext table became for x entries on wiby.me I want Wiby to be comprised mainly of human submitted pages, so for 99% of the index, only the pages submitted by users are indexed and no further crawling was done. However I recognized that not having the capability to crawl through links would not make it useful for others, so I added in the crawling capability to my liking and tested it accordingly.…
Short of having a private beta like kagi, how else could those bot nets be excluded? How difficult is it to create a white list of uninfected IPs?
clean simple code . is it used in production any where ?
Wiby is amazing and one of my favorite ways to discover the more unique side of the internet. I love to kill time by clicking "Surprise me" and half the RSS feeds I subscribe to are from this! Thank you so much for making Wiby! I have truly enjoyed what its brought!!