Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

news.ycombinator.com

Post reply on HN
Print view View original

421–430 of 492 posts

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#421

Post by mandeepj »

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

> Hardware costs are too high

I want to say - you don’t know what are talking about. But, it’ll be rude.

Hardware is much cheaper and powerful now compared to 2005.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#422

Post by ma2rten »

ampersandy wrote: post #243
ma2rten wrote: post #241

Earlier quoted context omitted.

It's much more expensive now to build a large index (50B+ pages) Do you have a cost estimate? Also could you be more selective in indexing, e.g. by having users requests sites to be crawled.

Requiring users to know what sites they want in advance somewhat defeats the purpose of a search engine, no?

More often than not I have an idea which site a result might be on when I issue a query:

If I search for a news event it's a news site.

If I search an error message, I know the result is going to likely be stackoverflow, github issues or the forum of the library.

etc.

I don't think this strategy will get you all the way there, but it could be combined with other ways of curating sites to crawl.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#423

Post by kf6nux »

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

What are your sources for hostnames to crawl?

I looked into it a long time ago and seem to remember there was a way to get access to registration records, but I imagine combining that with HTTP certificate transparency records would significantly increase your hostname list. Anything else?

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#424

Post by cphoover »

gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

what heuristics or AI is being used for blocking your spider? If your spider appears human or organuc it will not be blocked right?

Is this an issue of rate limiting, or request cadence? could you add randomness to the intervals in which you request the page?

Is it more complicated? do they use other signals to ascertain if you are a script or not like checking data from the browser (similar signals to the kind of things browser fingerprinting uses... e.g. screen res, user agent, cache availability, etc...) would it be possible for the browser to spoof this information?

I imagine rate limiting the IP address is the major issue... but could you not bounce the request through a proxy network? I've tried this with the TOR network before when writing web scrapers and had mixed success... seems like Google knows when a request is being made through Tor.

Perhaps you could use the users of your search engine as a proxy network through which to bounce the request for the scrape/indexing... This way the requests would look like they were coming from any of your users instead of one spiders ip address...Im not sure how cloudflare or any other reverse proxy could determine that thise requests were organic or not...

id be ok with contributing to a distributed search service so long as my cpu was not making requests to illegal content, and there were constraints put on the resource usage of my machine.

Sorry if this came off as all over the place, I do not know too much about the offense vs defense of scraping. These are just some thoughts...

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#425

Post by collin128 »

web007 wrote: post #167
collin128 wrote: post #132

Earlier quoted context omitted.

Have you ever looked at the Amazon file? I'll see if I can track down the link but I remember somebody sharing a dump with me from Amazon that apparently was a recent scrape. Edit: https://registry.opendata.aws/commoncrawl/

That's Common Crawl, they do the spidering of some billions of webpages but that's still a tiny percentage of the web versus Google or Bing.

Oh interesting, I've played with it a little but not a dev and I've always wondered what the coverage was like.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#426

Post by BbzzbB »

mandeepj wrote: post #421
gbmatt wrote: post #12

Ha, yes, I've done that at https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms…

> Hardware costs are too high I want to say - you don’t know what are talking about. But, it’ll be rude. Hardware is much cheaper and powerful now compared to 2005.

You've said it and it is rude, what's the point of that first sentence except to spite him? I'm sure he's well aware of the price per capability trend since 2005, you don't code a search engine without knowing that. Could be the costs of servicing his free users and/or maintaining an ever-growing database/index that is costly - in spite of cheaper hardware on a relative basis.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#427

Post by BbzzbB »

yellowstuff wrote: post #271
kilburn wrote: post #249

Earlier quoted context omitted.

I don't know about others, but when I think of the "good old google days" I'm _not_ expecting the results for your example queries to be any good. In those days querying took some effort but the effort paid off. The results for "history" just couldn't matter less in this mindset. You search for "USA history" or "house commons history" or "lake whatever history" instead. If the results come up with unexpected things m…

I get what you mean, but part of the whole initial appeal of Google was that it gave much more relevant results initially than Altavista or the other options. That was why Google put in the audacious "I'm feeling lucky" button.

>"I'm feeling lucky" button

My brain got so used to ignoring it I completely forgot it's a thing. I'm also unclear what it does? On an empty request, it gets me to their doodles page and with text in the box, gets me to my account history landing page.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#429

Post by astrange »

AtlasBarfed wrote: post #234
JPKab wrote: post #19

Earlier quoted context omitted.

Regarding the Gatekeeper companies like Cloudflare, it sounds like anti-competitive behavior that could potentially be targeted with anti-trust legislation, correct?

"targeted with anti-trust legislation" Um, this is America. Every market is basically a trust, cartel, or monopoly. And I don't know if you can hear that, but there is literally laughter in the halls of power. All the show hearings by congress on social media and tech companies only has to do with two things: 1) one political party thinking the other is getting an advantage by them 2) shaking them down for more lobby…

Companies don't make campaign donations. The people "exposing" them are showing their employees making donations, and employees don't have the same interests as their employer.

Re: Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

#430

Post by TechBro8615 »

xwdv wrote: post #308
GordonS wrote: post #82

Earlier quoted context omitted.

Heh, I guess you mean "trawling" - trolling the entire web is something very different :)

What would trolling the entire web look like?

It would look like a modern search engine with innovative technology offerings like Advanced Mobile Pages.
Post reply on HN