Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

news.ycombinator.com

Post reply on HN
Print view View original

301–310 of 492 posts

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#301

Post by skyde »

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

what kind of index is Gigablast using? traditional inverted index like Lucene or something more esoteric?

I know Google and Bing both use weird data-structure like BitFunnel

https://www.microsoft.com/en-us/research/publication/bitfunn...

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#302

Post by Nasrudith »

lolinder wrote: post #290

The consistent theme every time this comes up is that dealing with the sheer weight of the internet is almost impossible today. SEO spam is hard to fight and the index gets too heavy. However, I wonder if this is a sign that we're looking at the problem wrong. What if instead of even trying to index the entire web, we moved one step back towards the curated directories of the early web? Give users a search engine and…

The retro idea of curation seems popular here but everybody forgets why it lost out in the first place. It just doesn't scale in the first place. Not to mention demands - people usually want tools which lower mental effort and are intuitive as opposed to ones which are precise but in an obtuse metric. Most would not find a hardware mouse that consisted of two keypads for X and Y coordinates and a left click and right click button very useful.

Similarly everyone maintaining your their own index is cumbersome overkill in redundancy, processing power, and human effort in return for a stunted network graph which is worse for all metrics people usually actually care about. In terms of catching on even "antipattern search engines" that attempt to create an ideological echo chamber would probably catch on better.

Short of search engine experiments/start up attempts the only other useful application I can see is "rude web-spidering" which deliberately disrespects all requests to not index pages left publicly accessible as search engines generally try not to be good tools for cracker wardriving for PR and liability reasons. It would be a good whitehat or greyhat tool as doors secured by politeness only are not secure.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#303

Post by lolinder »

lessname wrote: post #297
lolinder wrote: post #290

The consistent theme every time this comes up is that dealing with the sheer weight of the internet is almost impossible today. SEO spam is hard to fight and the index gets too heavy. However, I wonder if this is a sign that we're looking at the problem wrong. What if instead of even trying to index the entire web, we moved one step back towards the curated directories of the early web? Give users a search engine and…

This might work well in some situations (e.g. research, development), however it would also increase the effect of echo chambers I think.

Possibly, but I'm not convinced.

Google's not exactly working against the echo chamber problem, and I think that's because to do so would be to work against its own reason for existing. There are two goals here that are fundamentally at odds with each other:

1) Finding what you're looking for.

2) Finding a new perspective on something.

A search engine's job is to address the first challenge: finding something that the user is looking for. The search engine might end up serving both needs if they're looking for a new perspective on something, but if these two goals ever come into conflict with each other the search engine does (and I would argue it should) choose the first goal. Failing to do so will just lead to people ignoring the results.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#304

Post by pythux »

1_player wrote: post #285
jermaustin1 wrote: post #281

Earlier quoted context omitted.

You are going to pay for a generalized web search when DDG/Google/Bing/etc are free?

Yes. I use Brave Search and I hope they add a paid tier, which I think they have confirmed they'll add at a later date. If you don't pay, you are the product. Simple as that.

https://twitter.com/brave/status/1466510541128548362?s=20

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#305

Post by xwdv »

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

How much cash do you need?

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#306

Post by mclightning »

lolinder wrote: post #290

The consistent theme every time this comes up is that dealing with the sheer weight of the internet is almost impossible today. SEO spam is hard to fight and the index gets too heavy. However, I wonder if this is a sign that we're looking at the problem wrong. What if instead of even trying to index the entire web, we moved one step back towards the curated directories of the early web? Give users a search engine and…

That's basically what I'm doing with my search "site:reddit.com" I wonder if anyone at Google is aware of this trend and taking notes.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#307

Post by entropie »

gbmatt wrote: post #205
DavidCole1 wrote: post #187

Earlier quoted context omitted.

Interesting. I had some interests in building a search engine myself (for playing around ofcourse). I had read a blog post by Michael Nielson [1] which had sparked my interest. Do you have any written material about your architecture and stuff like that? Would love to read up. [1]: https://michaelnielsen.org/ddi/how-to-crawl-a-quarter-billio...

there's some stuff here : https://github.com/gigablast/open-source-search-engine

Holy, thats a huge codebase. Github even shows no code/syntax hl for many cpp files because they are so big.

I fiddled around and searched for some not so well known sites in germany and the results were surprisingly good. But it looks really... aged.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#308

Post by xwdv »

GordonS wrote: post #82
thoughtstheseus wrote: post #31

Earlier quoted context omitted.

Perhaps trolling the entire web is not useful today? I’d love a search engine where I can whitelist sites or take an existing whitelist from trusted curators.

Heh, I guess you mean "trawling" - trolling the entire web is something very different :)

What would trolling the entire web look like?

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#309

Post by WalterBright »

I'd like to see categories like travel, science, history, art, etc. The web pages could pick which categories their page falls into using meta tags. The user has the option of selecting which category they are interested in searching within.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#310

Post by 1_player »

duckmysick wrote: post #293
1_player wrote: post #285

Earlier quoted context omitted.

Yes. I use Brave Search and I hope they add a paid tier, which I think they have confirmed they'll add at a later date. If you don't pay, you are the product. Simple as that.

> If you don't pay, you are the product. If not enough people pay, there's no product.

If nobody pays, there's even less of it. Not sure what's your point.
Post reply on HN