Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 492 posts

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#161

Post by jeffbee »

abhaynayar wrote: post #154

I'm probably the only person who doesn't think Google search has deteriorated. I play security CTFs, so a lot of times I have to search for peculiar technical details on various software. Also, like any other human being, I also make generic queries. In both cases, I feel like I almost always get to the desired webpage within the top few results.

Well no, it’s you and me and the whole google search quality evaluation team and everyone who works on google search and like 99% of the general public as well. The meme of falling search quality infects only HN. Mostly what people are complaining about is that the quality of the web itself is in free-fall.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#162

Post by Sohcahtoa82 »

jquery wrote: post #87
ErrrNoMate wrote: post #44

Earlier quoted context omitted.

Bitcoin and DeFi don't have utility outside of gambling and pump and dumps. Not everything (tbh not really anything) needs crypto.

Crypto’s biggest achievement is being the financial equivalent of the gulf war oil fires. Just massive pollution. Think of all the good things that computing could be used for… we used to have all kinds of interesting collaboration projects. Instead we are setting those CPU cycles on fire for short term profit.

Imagine if all that processing power was used for Folding@Home.

The problem is that cryptocurrencies do not inherently need tons of processing power to operate. You could theoretically run the entire Bitcoin network on a Raspberry Pi. But the PoW algorithm was designed to always produce a block every 10 minutes, no matter how much hashing power was dedicated to the network. Everyone wanted a piece of the block reward pie, so the arms race was created.

Proof-of-stake algorithms would eliminate this problem entirely, but PoS is a shitty "rich get richer" method. Granted, with how expensive mining power is, even PoW results in the rich getting richer, but at least it doesn't result in the wasting of gigawatts of electricity.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#163

Post by ramboldio »

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

maybe just add small webpages into your index, dont bother yo execute JS and dont download any images.

The content quality will be higher and it's a lot cheaper.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#164

Post by tigerlily »

teddyh wrote: post #135
tigerlily wrote: post #116

Surely there must be some way to have distributed search compute a la folding/seti@home or those mersenne prime guys. I'd gladly pool in some of my CPU time if it helps build a better search.

https://yacy.net/

Thanks!

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#165

Post by prox »

ColinHayhurst wrote: post #71
vincent_s wrote: post #9

Some people try: https://www.mojeek.com/ https://fireball.com/ https://search.brave.com/

Mojeek founder story here: https://blog.mojeek.com/2021/03/to-track-or-not-to-track.htm... No-tracking and independent from the start. Now at 4.6 billion pages with own infrastructure and IP. Went to market in 2020 with contextual ads and API. Self-disclosure: CEO

Never heard of Mojeek. I will try it for a month and see how it works. Currently using DDG 99% of the time.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#166

Post by rasengan »

snarkypixel wrote: post #106
rasengan wrote: post #2

This is how Private Search [1] works since it decouples the search from the user. This means nobody knows both who searched and what they searched for. This is a huge leap for privacy in search. [1] https://private.sh

Is it a proxy to other search engines or are they building their own?

It's a multi part partnership with Gigablast. Gigablast sees the searches, but not who searches. Private.sh sees who searches, but not what they search for.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#167

Post by web007 »

collin128 wrote: post #132
gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

Have you ever looked at the Amazon file? I'll see if I can track down the link but I remember somebody sharing a dump with me from Amazon that apparently was a recent scrape. Edit: https://registry.opendata.aws/commoncrawl/

That's Common Crawl, they do the spidering of some billions of webpages but that's still a tiny percentage of the web versus Google or Bing.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#168

Post by jeffbee »

willcipriano wrote: post #151

I'd like to see a "just search" engine, all it does it search for a specific string, case insensitively, across the entire web. No curation or anything, just sorted in lexicological order closest match first maybe falling back to page age if it has more then one exact match. Perhaps give me some regular expressions as well.

That would be easily the worst search engine ever deployed. Imagine just returning all docs containing the word “bicycle” in chronological order. Useless.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#169

Post by MayeulC »

DavideNL wrote: post #100
nfriedly wrote: post #59

I think DuckDuckGo is closer to what you want. Same results for everyone, better privacy, and they're proactive about improving their results. https://duckduckgo.com/ Part of the problem is that there's a lot more low-quality content to wade through now than there was in 2005. I think the Google of 2005 would have trouble delivering quality results today also.

> a lot more low-quality content I wish there was an easy way to filter ALL search results, by permanently excluding specific websites, and/or keywords. Surely there has to be some browser extension that does this...

Excluding, or penalizing for, advertising and trackers could do wonders against perverse incentives and SEO, IMO. It would also be a better experience for the reader.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#170

Post by gk1 »

BitwiseFool wrote: post #24
marginalia_nu wrote: post #17

Earlier quoted context omitted.

I think some NLP is strictly beneficial for a search engine. You may think "grep for the web" sounds like a good idea, but let me tell you, having tried this, manually going through every permutation of plural forms of words and manually iterating the order of words to find a result is a chore and a half. Like, instead of trying PDP11 emulator PDP-11 emulator "PDP 11" emulator PDP11 emulators PDP-11 emulators "PDP 11…

Maybe I'm not using the right qualifiers around the term NLP. The kind of NLP I was referring to is something like "Hey google, what is natural language processing?" and orienting the search around people asking questions in standard(ish) English like they would to another person.

That's known as Open Domain Question Answering[1] and is only a subset of NLP.

[1] https://www.pinecone.io/learn/question-answering/

Post reply on HN