Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The world needs a non-profit search engine

daoudclarke.net

Post reply on HN
Print view View original

41–50 of 396 posts

Re: The world needs a non-profit search engine

#41

Post by thatwasunusual »

tyropita wrote: post #35
thatwasunusual wrote: post #26

Earlier quoted context omitted.

I have also thought that distributed crawling with the help of browser extensions, and/or clients like folding@home, could be a good idea. But how to deal with "spam injections"?

After a certain scale I think you can let clients do double-work and let the most common crawl data, among different clients, win. And since you control what URLs need to be crawled, you protect yourself against rogue clients sending arbitrary URLs. There certainly are a lot of elegant ways to reduce spam for this particular problem imo.

> And since you control what URLs need to be crawled, you protect yourself against rogue clients sending arbitrary URLs.

I'm not worried about the URLs, but the content of the URLs sent back.

Say the server tells a client to crawl a CNN article. The "hacked" client sends a fake CNN article back.

Re: The world needs a non-profit search engine

#42

Post by defrost »

eterevsky wrote: post #27

> Google tries to work out which sites are interesting by how long you spend on the site. How would Google know how long you spend on the site? It only sees what links you clicked and doesn't know what happens next. (Unless the website uses Analytics, but Analytics doesn't affect search ranking.)

Let's not forget they also have a very popular browser that itself collects and bundles back home a lot of usage infomation from most of its user base:

https://www.ghacks.net/2021/03/16/wonder-about-the-data-goog...

Re: The world needs a non-profit search engine

#43

Post by unicornporn »

irrational wrote: post #6

I use duck duck go. Recently someone showed me their screen where they were using google to do a search. I was absolutely aghast. The last time I used google when you searched for something you saw a simple text list of sites (which is how DDG still works). Instead the google results were… a disaster. You had to scroll through some much garbage before finding actual search results - a list of sites. It was like googl…

> I use duck duck go.

You use a proxied Bing. If you want a proxied Google (what I prefer), you can use https://startpage.com

Re: The world needs a non-profit search engine

#44

Post by neoglow »

rvz wrote: post #36
irrational wrote: post #6

I use duck duck go. Recently someone showed me their screen where they were using google to do a search. I was absolutely aghast. The last time I used google when you searched for something you saw a simple text list of sites (which is how DDG still works). Instead the google results were… a disaster. You had to scroll through some much garbage before finding actual search results - a list of sites. It was like googl…

Who's going to finally tell them who DDG really relies on?

I get your point, but the fact remains that the interface is miles ahead of Google, just by not having all the garbage.

The results are fine for me in day to day usage and I find that Google will not provide me with better results if I cant find it on DDG.

Would it be better if they had their own indexer? Maybe...

Re: The world needs a non-profit search engine

#45

Post by noisenotsignal »

As the underlying project discussed by this post is a search engine, I searched for “mwmbl” on mwmbl.org [0], and no results were found! Relevant results like the main site and GitHub repo show up when searched on Google or Kagi.

[0] - https://mwmbl.org/?q=mwmbl

Re: The world needs a non-profit search engine

#47

Post by CobrastanJorji »

noisenotsignal wrote: post #45

As the underlying project discussed by this post is a search engine, I searched for “mwmbl” on mwmbl.org [0], and no results were found! Relevant results like the main site and GitHub repo show up when searched on Google or Kagi. [0] - https://mwmbl.org/?q=mwmbl

I searched for "Google" on mwmbl, and while the first page of results found many results, including Google Patents, Google Bug Hunters, and a Google Books page on a 2004 book about Anarchism, it did not find Google's home page.

I searched for "Elephant," and I got a Wikipedia page about a specific Elephant statue at Coney Island, a UK elephant charity, and a blog post about Haskell ("the elephant in the room").

It's unfair to poke fun at a very small project that admits that it is far from done yet, but it's gotta figure out a way to crack the "which pages are most likely to be relevant" problem or else it's not going to be useful.

Re: The world needs a non-profit search engine

#49

Post by daoudc »

noisenotsignal wrote: post #45

As the underlying project discussed by this post is a search engine, I searched for “mwmbl” on mwmbl.org [0], and no results were found! Relevant results like the main site and GitHub repo show up when searched on Google or Kagi. [0] - https://mwmbl.org/?q=mwmbl

Yeah the search quality has a way to go!

Re: The world needs a non-profit search engine

#50

Post by FridgeSeal »

unicornporn wrote: post #43
irrational wrote: post #6

I use duck duck go. Recently someone showed me their screen where they were using google to do a search. I was absolutely aghast. The last time I used google when you searched for something you saw a simple text list of sites (which is how DDG still works). Instead the google results were… a disaster. You had to scroll through some much garbage before finding actual search results - a list of sites. It was like googl…

> I use duck duck go. You use a proxied Bing. If you want a proxied Google (what I prefer), you can use https://startpage.com

I believe a few years ago startpage was sold to an advertising company.
Post reply on HN