Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Wiby is now free software

github.com

Post reply on HN
Print view View original

1–10 of 36 posts

Re: Show HN: Wiby is now free software

#5

Post by rav3ndust »

This is awesome - thanks for sharing. I just so happen to have a domain left lying around for the Searx instance I never spun up, so I might end up using it for a weekend project with this.

Love Wiby, been playing with it awhile now. I especially enjoy the "surprise me" button from time to time.

Re: Show HN: Wiby is now free software

#7

Post by flas9sd »

the guide linked in the Readme is comprehensive and a good read - also gives instructions on who to scale and distribute load.

If the author can answer: how big did the fulltext table became for x entries on wiby.me - and what is a common response time on N amount of searches per minute for this dataset?

Would you offer a /traffic or /stats page within about/ ? duckduckgo shows traffic, not index stats though.

I don't see it in the windex schema yet, but would be interesting what the actual hitrate is on a search corpus, how many clickthroughs there are for any used search term. Answering these kind of questions adds computation and record keeping though.

Thanks for open sourcing, it's an interesting mix of languages and tools!

Re: Show HN: Wiby is now free software

#8

Post by wibyweb »

flas9sd wrote: post #7

the guide linked in the Readme is comprehensive and a good read - also gives instructions on who to scale and distribute load. If the author can answer: how big did the fulltext table became for x entries on wiby.me - and what is a common response time on N amount of searches per minute for this dataset? Would you offer a /traffic or /stats page within about/ ? duckduckgo shows traffic, not index stats though. I don'…

>how big did the fulltext table became for x entries on wiby.me

I want Wiby to be comprised mainly of human submitted pages, so for 99% of the index, only the pages submitted by users are indexed and no further crawling was done. However I recognized that not having the capability to crawl through links would not make it useful for others, so I added in the crawling capability to my liking and tested it accordingly. I imagine others might want to depend heavily on hyperlink crawling for their use case, but there is a tradeoff in the quality of the pages that get indexed and the resources they require.

>and what is a common response time on N amount of searches per minute for this dataset?

Hard to say exactly as I haven't run many benchmarks, but my goal is to keep multi-word queries to within about a second. Single-word queries are very fast. My 4 computers handle hundreds of thousands of queries per day because Wiby is being barraged by a nasty spam botnet with thousands of constantly changing IPs. If I don't keep them in check they will eventually eat all the CPU availability.

>Would you offer a /traffic or /stats page within about/ ? duckduckgo shows traffic, not index stats though.

Probably not on mine since I don't get enough traffic for it to be of that much interest to me. I privately use goaccess to get a general idea of daily traffic.

Re: Show HN: Wiby is now free software

#9

Post by wibyweb »

rambambram wrote: post #3

Indeed interesting! I might have missed it, but is this 'only' for building an index? Or also for searching through the results in some way? For now, I'm definitely adding wiby.me to my list of recommended search engines.

With this code you will start out with a blank index and will have to start making submissions to your search engine to build the index, but you can search the results as soon as the pages get crawled. The video demo provides a practical example.

Re: Show HN: Wiby is now free software

#10

Post by wibyweb »

marginalia_nu wrote: post #2

Good luck, and interesting approach! How large is your index, by the way, like in terms of documents and gigabytes?

Thank you, my index is puny compared to yours and that is because I only index the page submitted by guests and don't go any further. The index is in the 10's of thousands. The hyperlink crawling feature was added and tested out specifically for the release, as I understand that some people will want to use that heavily and to build up a much larger index. My computers were super cheap and handle my puny index well, because its puny. I have no idea how well my approach compares to others in terms of performance.
Post reply on HN