Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 492 posts

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#121

Post by JohnFen »

infinitezest wrote: post #85
KennyBlanken wrote: post #46

Earlier quoted context omitted.

For me, DDG results are even worse than Google. It's set as my default and I'd say at least half of my searches in DDG generate completely useless results...pages of obviously SEO'd garbage. DDG also doesn't support showing a site's basic structure in the search results (ie, the card of a company's website with Products, Contact Us, Support, etc) and the preview text is garbo as well...it reminds me of 1990's era ele…

Every time this comes up I'll see a few people talk about how the results aren't relevant but it has not been my experience. I've been using DDG as my main search engine for a few years and never have to go beyond the first page. I really curious why that is.

My experience is like yours -- DDG is legitimately better than Google. My hypothesis is that it's related to how you construct searches. I expect Google probably does better if you learn how to talk to it, since it seems to want to interpret your query rather than take it literally.

My searches tend to be keyword-oriented rather than natural language. I think DDG does better with those.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#122

Post by marginalia_nu »

thefreeman wrote: post #101
ginko wrote: post #30

Earlier quoted context omitted.

The point is that SEO would just immediately adapt to Google's changes. If a smaller search engine filtered these out then it would likely stay under the radar.

you know that legitimate sites perform SEO as well, right?

SEO often seems to be a compensation for the fact that a site doesn't have particularly worthwhile content. So punishing SEO surprisingly does promote higher quality of search results.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#123

Post by adolph »

JPKab wrote: post #19
gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

Regarding the Gatekeeper companies like Cloudflare, it sounds like anti-competitive behavior that could potentially be targeted with anti-trust legislation, correct?

At a theoretical level it looks like Cloudflare won't block search engine crawlers. The docs are very Google and Bing oriented and also oriented towards supporting their customers, not random new search engine crawler.

Cloudflare allows search engine crawlers and bots. If you observe crawl issues or Cloudflare challenges presented to the search engine crawler or bot, contact Cloudflare support with the information you gather when troubleshooting the crawl errors via the methods outlined in this guide.

https://support.cloudflare.com/hc/en-us/articles/200169806

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#124

Post by BbzzbB »

marginalia_nu wrote: post #32
BbzzbB wrote: post #26

Earlier quoted context omitted.

Marginalia is great to find blog posts, personal sites and other long form content, but it's not a replacement for Google nor intends to.

It does operate on a scale and principle fairly similar to early 2000s google, so the comparison isn't that far off, but yeah, it's quite some way before it's viable for general search. Dunno if I'll ever get there, but it does consistently seem to get better so who knows.

Isn't it's familiarity to early Google a side-effect of the early Internet being text-heavy sites in the first place rather than a similarity in the search engine? Unless I am misunderstanding your site's intent, even if you reach the dream engine you are trying to achieve, I won't be using it to search answers for coding questions on SO, how-tos for car repair, sites to stream movies, governmental page for X need, transcript for earnings calls, etc.

In my experience it is better than Google at what it does if I'm looking for long-form texts (exception being scientific/peer-reviewed articles, Google tends to shoot me those for the type of queries I make on Marginalia), but is very much complementary rather than a replacement.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#125

Post by loo »

garaetjjte wrote: post #94
gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

>Gigablast has teamed up with Imperial Family Companies Associating with that crank (responsible for recent freenode drama) is very off-putting.

Oh no, you see he isn't responsible, it's everyone else! /s

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#126

Post by marginalia_nu »

Zamicol wrote: post #74
erhk wrote: post #38

Earlier quoted context omitted.

Trusted curators is a dangerous dependency

How about just a meritocratic rating? Even here on HN I would appreciate some sort of weight on expert/experienced opinion. Although in theory I like the idea that every thought is judged on its own, the context of the author is more relevant the deeper the subject. That's one of the reasons I still read https://lobste.rs . It has a niche audience with industry experience.

> meritocratic rating

That is literally PageRank.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#127

Post by gbmatt »

mirker wrote: post #51
gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

If you have customers, does that mean the incremental gain from an improved index costs too much to store? Or are you talking about computational costs?

it's both storage and computational. they go hand in hand.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#128

Post by foxfluff »

technobabbler wrote: post #48
JPKab wrote: post #19

Earlier quoted context omitted.

Regarding the Gatekeeper companies like Cloudflare, it sounds like anti-competitive behavior that could potentially be targeted with anti-trust legislation, correct?

Cloudflare functions kinda like a private security company. They don't go around blocking sites willy-nilly, site owners have to specifically choose to use their service (and maybe pay for it), configuring the bot blocking rules themselves. That's not really Cloudflare's fault. Someone has to do it, whether it's them or a competitor or sys admins manually making firewall rules. Cloudflare just happens to be good enou…

No, I think it is partially Cloudflare's fault because they offer this service and make it easy to deploy. This shit has exploded with Cloudflare's popularity.

Nobody has to do it, but a lot of people will do it when they notice there's an easy way to do it. Cloudflare is very much an enabler of bad behavior here. Now a lot of sites just toggle that on without even thinking about collateral.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#129

Post by technobabbler »

gbmatt wrote: post #80
technobabbler wrote: post #48

Earlier quoted context omitted.

Cloudflare functions kinda like a private security company. They don't go around blocking sites willy-nilly, site owners have to specifically choose to use their service (and maybe pay for it), configuring the bot blocking rules themselves. That's not really Cloudflare's fault. Someone has to do it, whether it's them or a competitor or sys admins manually making firewall rules. Cloudflare just happens to be good enou…

I've had extensively dealing with Cloudflare. They have a complex whitelisting system that is difficult to get on, and they also have an 'AI' system that determines if you should be kicked off that whitelist for whatever reason. Furthermore, they give Google preferred treatment in their UIs and backend algos because it is the incumbent and nobody cares about other smaller search engines. So there's a lot of detail to…

These are all great points.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#130

Post by adolph »

hunterb123 wrote: post #90
nfriedly wrote: post #59

I think DuckDuckGo is closer to what you want. Same results for everyone, better privacy, and they're proactive about improving their results. https://duckduckgo.com/ Part of the problem is that there's a lot more low-quality content to wade through now than there was in 2005. I think the Google of 2005 would have trouble delivering quality results today also.

DDG never worked great for me, and it doesn't have it's own index. Brave search has been my daily driver and it works wonderful.

I'll give it a try, somehow I missed the announcement even though I'm a Brave user...

https://brave.com/search/

Post reply on HN