Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Wiby is now free software

github.com

Post reply on HN
Print view View original

31–36 of 36 posts

Re: Show HN: Wiby is now free software

#31

Post by Komodai »

wibyweb wrote: post #14
Komodai wrote: post #4

Wow! Thank you so much for this, I will for sure try and play around with this. Is it possible to run the Crawler behind like socks5 proxy or such?

You can certainly change the crawler's database connection from "localhost" to an IP address on a different machine, but I am unsure how that works with that type of proxy (I had to look up what a socks5 is). Sounds like it can work though.

Well what I meant was, say I have a server with the ip 13.223.12.212 and I want to run the crawler there, however would like to crawl the actual websites with the ip 23.215.23.15 (aka my proxy, socks5 being one of the several protocols to do it)

If you get what i mean :P

I assume it's possible if I just change some of the curl options in the crawler code

Re: Show HN: Wiby is now free software

#32

Post by wibyweb »

prox wrote: post #22
wibyweb wrote: post #8

Earlier quoted context omitted.

>how big did the fulltext table became for x entries on wiby.me I want Wiby to be comprised mainly of human submitted pages, so for 99% of the index, only the pages submitted by users are indexed and no further crawling was done. However I recognized that not having the capability to crawl through links would not make it useful for others, so I added in the crawling capability to my liking and tested it accordingly.…

What is the botnet owner thinking of gaining from a small potatoes search engine? Seems rather futile?

I wish I knew. They have nothing to gain. Its effectively a DDoS attack.

Re: Show HN: Wiby is now free software

#33

Post by wibyweb »

Komodai wrote: post #31
wibyweb wrote: post #14

Earlier quoted context omitted.

You can certainly change the crawler's database connection from "localhost" to an IP address on a different machine, but I am unsure how that works with that type of proxy (I had to look up what a socks5 is). Sounds like it can work though.

Well what I meant was, say I have a server with the ip 13.223.12.212 and I want to run the crawler there, however would like to crawl the actual websites with the ip 23.215.23.15 (aka my proxy, socks5 being one of the several protocols to do it) If you get what i mean :P I assume it's possible if I just change some of the curl options in the crawler code

Do let me know if you succeed

Re: Show HN: Wiby is now free software

#35

Post by em-bee »

wibyweb wrote: post #8
flas9sd wrote: post #7

the guide linked in the Readme is comprehensive and a good read - also gives instructions on who to scale and distribute load. If the author can answer: how big did the fulltext table became for x entries on wiby.me - and what is a common response time on N amount of searches per minute for this dataset? Would you offer a /traffic or /stats page within about/ ? duckduckgo shows traffic, not index stats though. I don'…

>how big did the fulltext table became for x entries on wiby.me I want Wiby to be comprised mainly of human submitted pages, so for 99% of the index, only the pages submitted by users are indexed and no further crawling was done. However I recognized that not having the capability to crawl through links would not make it useful for others, so I added in the crawling capability to my liking and tested it accordingly.…

i like this approach as a possible use for a personal searchengine, that only has stuff that i have been looking at. for that it would be helpful to have some kind of browser extension that can autosubmit everything in my history. ideally that extension would also autoaccept every submission so that it can work fully in the background without my intervention.

also helpful would be a whilelist/blacklist feature, say, wikipedia and stackoverflow may always be autoaccepted while certain other sites may always be rejected, and the rest go through the regular review process.

then i can use that as my default search engine and branch out when i don't find what i am looking for. for that it would also be cool if there could be a way to search wiby and another search engine in paralell and display like 5 results from each.

Re: Show HN: Wiby is now free software

#36

Post by wibyweb »

em-bee wrote: post #35
wibyweb wrote: post #8

Earlier quoted context omitted.

>how big did the fulltext table became for x entries on wiby.me I want Wiby to be comprised mainly of human submitted pages, so for 99% of the index, only the pages submitted by users are indexed and no further crawling was done. However I recognized that not having the capability to crawl through links would not make it useful for others, so I added in the crawling capability to my liking and tested it accordingly.…

i like this approach as a possible use for a personal searchengine, that only has stuff that i have been looking at. for that it would be helpful to have some kind of browser extension that can autosubmit everything in my history. ideally that extension would also autoaccept every submission so that it can work fully in the background without my intervention. also helpful would be a whilelist/blacklist feature, say,…

Perhaps you can develop such a browser extension. Sounds like a very good idea actually.
Post reply on HN