Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google? • Hacker News Board

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#41

Post by KennyBlanken » Thu, Dec 02, 2021, 3:43 PM UTC

marginalia_nu wrote: post #17

BitwiseFool wrote: post #4

Natural Language Processing is a pox on modern search engines. I suspect that Google et. al. wanted to transform their product into an answer engine that powers voice assistants like Siri and just assumed everyone would naturally like the new way better. I can't stand how Google is always trying to guess what I want, rather than simply returning non-personalized results solely based on exactly what I typed in the tex…

I think some NLP is strictly beneficial for a search engine. You may think "grep for the web" sounds like a good idea, but let me tell you, having tried this, manually going through every permutation of plural forms of words and manually iterating the order of words to find a result is a chore and a half. Like, instead of trying PDP11 emulator PDP-11 emulator "PDP 11" emulator PDP11 emulators PDP-11 emulators "PDP 11…

Google does go way overboard with "NLP". Starting at least 5 years ago there was a trend toward "similar" matching and search result quality nose-dived.

You can search for, say, "cycling (insert product category here)" and get motorcycle related results. Why? Because to google "cycling" = "biking" and "motorcycles" are "bikes", bob's your uncle, now you're getting hits for motorcycle products.

Every time I try to do a very specific search I can see from the search results how google tries to "help", especially if the topic is esoteric. The pages actually about the esoteric thing I'm searching for get drowned in a sea of SEO'd bullshit about a word/topic that is 1-2 degrees of separation from each other in a thesaurus. I'm sure someone at google is very, very proud of this because it increases their measure for search user satisfaction X percent.

It does this thesaurus crap even with words in quotes, which is especially infuriating.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#42

Post by amelius » Thu, Dec 02, 2021, 3:43 PM UTC

Also, where are the books about writing a search engine?

Knuth's "Searching and Sorting" volume desperately needs an update.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#43

Post by indymike » Thu, Dec 02, 2021, 3:44 PM UTC

Brave's new search engine seems to work pretty well. Have been using it as my primary for about 10 days, and so far, I've only had to revert to Google once, and when I did the results were chock full of spam.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#44

Post by ErrrNoMate » Thu, Dec 02, 2021, 3:44 PM UTC

mrkramer wrote: post #29

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

I'm sorry to say but your project is 20 years old and it had no impact at all. You are doing something wrong. Innovation and initiative is needed ala Bitcoin and DeFi not hobby projects which are not picking up in popularity and utility.

Bitcoin and DeFi don't have utility outside of gambling and pump and dumps. Not everything (tbh not really anything) needs crypto.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#45

Post by gbmatt » Thu, Dec 02, 2021, 3:45 PM UTC

JPKab wrote: post #19

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

Regarding the Gatekeeper companies like Cloudflare, it sounds like anti-competitive behavior that could potentially be targeted with anti-trust legislation, correct?

it should be. there should be some sort of 'bots rights' to level the playing field. perhaps this is something the FTC can look into. but, as it is right now big tech continues to keep their iron grip on the web and i don't see that changing any time soon. big tech has all the money and controls access to all the data and supply chains to prevent anyone else from being a competitive threat.

look at linkedin (owned by microsoft unspiderable by all but google/bing). github (now microsoft using this to fuel its AI coding buddy, but if you try to spider this at capacity your IP is banned) facebook (unspiderable) .. the list goes on and on ..

and as you can see, data is required to train advanced AI systems, too. So big tech has the advantage there as well. especially when they can swoop in and corrupt once non-profit companies like openai, and make them [partially] for-profit.

and to rant on (yes, this is what i do :)) it very difficult to buy a computer now. have you tried to buy a raspberry pi or even a jetson nano lately? Who is getting preferred access to the chip factories? Does anyone know? Is big tech getting dibs on all the microchips now too?

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#46

Post by KennyBlanken » Thu, Dec 02, 2021, 3:46 PM UTC

BbzzbB wrote: post #22

No mention of DDG in the comments? Is there a reason I'm not seeing or it's just not the preferred alt-search on HN? Seems to have been working fine for me when I struggle to get past the funnels and content mills on Google.

For me, DDG results are even worse than Google. It's set as my default and I'd say at least half of my searches in DDG generate completely useless results...pages of obviously SEO'd garbage.

DDG also doesn't support showing a site's basic structure in the search results (ie, the card of a company's website with Products, Contact Us, Support, etc) and the preview text is garbo as well...it reminds me of 1990's era electronic card catalog search excerpts.

I look at the first page or two, give up, search google. While I have to hunt a bit in the results, I do eventually get what I wanted.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#47

Post by lloydatkinson » Thu, Dec 02, 2021, 3:46 PM UTC

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

With a slightly fresher coat of paint this could be very popular. For example, no grey background.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#48

Post by technobabbler » Thu, Dec 02, 2021, 3:46 PM UTC

JPKab wrote: post #19

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

Regarding the Gatekeeper companies like Cloudflare, it sounds like anti-competitive behavior that could potentially be targeted with anti-trust legislation, correct?

Cloudflare functions kinda like a private security company. They don't go around blocking sites willy-nilly, site owners have to specifically choose to use their service (and maybe pay for it), configuring the bot blocking rules themselves.

That's not really Cloudflare's fault. Someone has to do it, whether it's them or a competitor or sys admins manually making firewall rules. Cloudflare just happens to be good enough and darned affordable, so many choose to use them.

Hosting costs for small site owners would be much more expensive without Cloudflare shielding and caching.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#49

Post by thoughtstheseus » Thu, Dec 02, 2021, 3:47 PM UTC

erhk wrote: post #38

thoughtstheseus wrote: post #31

Earlier quoted context omitted.

Perhaps trolling the entire web is not useful today? I’d love a search engine where I can whitelist sites or take an existing whitelist from trusted curators.

Trusted curators is a dangerous dependency

It is. The alternative is scooping everything and using algos to curate. That seems worse imo.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#50

Post by swframe2 » Thu, Dec 02, 2021, 3:47 PM UTC

Have a look at gpt-3 if you want to see what the future dominant search engine will be. It will not find relevant results, it will write it on the fly customized for exactly what you want to read. (Maybe products will just ship to your door and be auto paid because the future ad targeting AIs will know you so well.)