Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The world needs a non-profit search engine

daoudclarke.net

Post reply on HN
Print view View original

31–40 of 396 posts

Re: The world needs a non-profit search engine

#31

Post by 6510 »

thatwasunusual wrote: post #26
tyropita wrote: post #25

Quite a neat way to crawl websites using a browser extension. That by itself is a form of donation to the search engine. Maybe in the future you can have dedicated software for self-hosted clients that users can run to crawl and index websites for mwmbl? Kinda like folding@home. How are the batches of URLs to be crawled generated/discovered and posted at your API? How do you deal with duplicate crawls?

I have also thought that distributed crawling with the help of browser extensions, and/or clients like folding@home, could be a good idea. But how to deal with "spam injections"?

YaCy just crawls the results [again] locally before showing them to you.

Re: The world needs a non-profit search engine

#32

Post by gnramires »

y42 wrote: post #9

I totally like the idea but I dare to doubt that this would solve the SEO problem. Website owners who are participating in those notorious affiliate programs or earn money with ads will still use the search engine to drag people onto their generic websites, using methodolgies to fit the search engines ranking mechanisms, no matter if they are public or not. SEO and all it's results seem to be immanent to the system.

You can have a semi-manual system, with user input. So websites filled with SEO get manually flagged by users. Of course, you need to trust your users as well... this can be done with a Web of Trust-style reputation system: users endorse each other, and you can build a reputation graph/tree that traces reputation and easily cull bad subtrees. If this endorsement system conserves reputation (you give a fraction of your reputation away each endorsement, and new reputation is never created), then it becomes sybil-proof, where it's not advantageous to create say millions of users to increase reputation.

(previous suggestion here: https://news.ycombinator.com/item?id=31585340)

Re: The world needs a non-profit search engine

#33

Post by thatwasunusual »

6510 wrote: post #31
thatwasunusual wrote: post #26

Earlier quoted context omitted.

I have also thought that distributed crawling with the help of browser extensions, and/or clients like folding@home, could be a good idea. But how to deal with "spam injections"?

YaCy just crawls the results [again] locally before showing them to you.

Maybe I misunderstand, but doesn't that mean you lose the benefit of having distributed crawlers if everything has to be crawled (again) locally somewhere?

Re: The world needs a non-profit search engine

#34

Post by mattrick »

One feature that I really wish more search engines would have is the ability to blocklist certain domains, particularly ones whose results are never relevant or helpful to the query itself (Pinterest, Quora, etc). It could even be used as a factor in the site’s search rankings.

Re: The world needs a non-profit search engine

#35

Post by tyropita »

thatwasunusual wrote: post #26
tyropita wrote: post #25

Quite a neat way to crawl websites using a browser extension. That by itself is a form of donation to the search engine. Maybe in the future you can have dedicated software for self-hosted clients that users can run to crawl and index websites for mwmbl? Kinda like folding@home. How are the batches of URLs to be crawled generated/discovered and posted at your API? How do you deal with duplicate crawls?

I have also thought that distributed crawling with the help of browser extensions, and/or clients like folding@home, could be a good idea. But how to deal with "spam injections"?

After a certain scale I think you can let clients do double-work and let the most common crawl data, among different clients, win.

And since you control what URLs need to be crawled, you protect yourself against rogue clients sending arbitrary URLs.

There certainly are a lot of elegant ways to reduce spam for this particular problem imo.

Re: The world needs a non-profit search engine

#36

Post by rvz »

irrational wrote: post #6

I use duck duck go. Recently someone showed me their screen where they were using google to do a search. I was absolutely aghast. The last time I used google when you searched for something you saw a simple text list of sites (which is how DDG still works). Instead the google results were… a disaster. You had to scroll through some much garbage before finding actual search results - a list of sites. It was like googl…

Who's going to finally tell them who DDG really relies on?

Re: The world needs a non-profit search engine

#39

Post by 6510 »

daoudc wrote: post #13
y42 wrote: post #9

I totally like the idea but I dare to doubt that this would solve the SEO problem. Website owners who are participating in those notorious affiliate programs or earn money with ads will still use the search engine to drag people onto their generic websites, using methodolgies to fit the search engines ranking mechanisms, no matter if they are public or not. SEO and all it's results seem to be immanent to the system.

I'm hoping that having community moderated search results will limit this problem. The problem will be SEO people trying to infiltrate the community, which may be tricky to solve. But Wikipedia has had to grapple with similar problems, so I think it is solvable.

Sell weekly SEO guids to rank well and punish last weeks tips. (joking)

What you want to do is put the SEO monster in front of your cart and make it do useful work. You basically got an army of hard working people with money to burn who will do anything. What is there to complaint about?

Re: The world needs a non-profit search engine

#40

Post by lukeschwartz »

benrmatthews wrote: post #38

How is the name of the site pronounced? “Mah-wim-ball”? Google’s name was so ubiquitous it became a verb, Duck Duck Go is a smart memorable name. Mwmbl is a challenging product name, even if the .org domain name was available.

https://github.com/mwmbl/mwmbl#frequently-asked-question
Post reply on HN