Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

news.ycombinator.com

Post reply on HN
Print view View original

481–490 of 492 posts

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#481

Post by carlesfe »

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

Great job, I didn't know aboug Gigablast and it looks very interesting. Can I give you a small piece of feedback? I just tried searching for myself on Gigablast, and the first results are profile pages which haven't been updated since like 2005. Meanwhile, my own personal page appears on the very bottom of the results.

So my suggestion would be to lower the weight of the ranking of the domain, and promote sites which have a more recent update date.

Send me an email (contact in profile) if you want to follow up on this feedback!

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#482

Post by BrendanEich »

gbmatt wrote: post #189
pythux wrote: post #171

Earlier quoted context omitted.

This is not correct. Brave Search owns its own (growing) index and relies on third-parties like Bing for some fraction of the requests. Which is not the same thing as relying fully on Bing or third-parties for results like so many meta-search engines. More detailed answer here: https://search.brave.com/help/independence Edit: Forgot to say that I work on Brave Search.

brave 'falls back' to bing. which in my experience is most of the time. in fact, out of all the queries i did a while back, they all seemed to come directly from bing. is there a way to disable the reliance on bing and get pure 'brave only' results? and can you be more specific as to what this fraction is? do you blend at all?

What independence percentage do you see when you click on the gear in upper right of the Brave Search results page?

I get 84% personal (browser-based), 87% global (which means we hit Bing only 13% of the time from our server side).

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#483

Post by BrendanEich »

diamondage wrote: post #456
1_player wrote: post #285

Earlier quoted context omitted.

Yes. I use Brave Search and I hope they add a paid tier, which I think they have confirmed they'll add at a later date. If you don't pay, you are the product. Simple as that.

Telegram, Signal, Mozilla are counterexamples... Have a large charitable donated cash balance sitting in your account, and your organisational motivation is all different

Mozilla Foundation does not fund Firefox, that's in an arms-length wholly owned for-profit subsidiary and Google is main source of funding via the search deal.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#485

Post by romwell »

giardini wrote: post #467
romwell wrote: post #267

Earlier quoted context omitted.

Well, no, it's not fine. See e.g. the source you linked , which explains the difference.

Did you read to the end? Methinks not!

>Did you read to the end? Methinks not!

Methink harder.

>Troll for means to patrol or wander about an area in search of something. Trawl for means to search through or gather from a variety of sources.

We were talking about gathering information from a variety of sources to build a search engine index.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#486

Post by _HMCB_ »

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

The Internet is such a fabric of society that I think all nations should contribute to a one-truth index. Not owned by a corporate entity. Tell me I’m wrong and we can consider the alternative: startups of all types with a more even playing field.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#487

Post by onecommentman »

llaolleh wrote: post #471
datenarsch wrote: post #223

Earlier quoted context omitted.

I wish I could agree but from my experience, DDG's search results aren't really that great. Often even worse than Google's. And another private company is not the answer I believe. We need something more drastic, an open-source search engine organized as a genuine non-profit organization. Something like that. Otherwise, whatever replaces Google will just turn into another Google as soon as it gets any momentum.

I think open source will be tough because you're going to need a lot of saints to work on a search engine of Google's caliber. Maybe an alternative revenue model instead of ads.

Consortium of universities, perhaps? Every top school (globally) kicks in some design and development time. It seems odd that the most critical link to access information on the planet is not the product of academia. With a country’s skin in the global game, there may be better leverage to keep it free and open for their citizens.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#489

Post by rnotaro »

cphoover wrote: post #424
gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

what heuristics or AI is being used for blocking your spider? If your spider appears human or organuc it will not be blocked right? Is this an issue of rate limiting, or request cadence? could you add randomness to the intervals in which you request the page? Is it more complicated? do they use other signals to ascertain if you are a script or not like checking data from the browser (similar signals to the kind of th…

> I've tried this with the TOR network before when writing web scrapers and had mixed success... seems like Google knows when a request is being made through Tor.

That's because all the TOR entry/exit nodes and relays IPs addresses are public [1].

[1] https://metrics.torproject.org/rs.html#toprelays

Post reply on HN