Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only Google is really allowed to crawl the web

knuckleheads.club

Post reply on HN
Print view View original

101–110 of 365 posts

Re: Only Google is really allowed to crawl the web

#101

Post by skinkestek »

jinseokim wrote: post #19

This has been submitted to HN quite a few times. https://news.ycombinator.com/item?id=25426662 (Most comments; 11 comments) https://news.ycombinator.com/item?id=25417067 (3 comments) https://news.ycombinator.com/item?id=25546867 (Most recent; 89 days ago) https://news.ycombinator.com/item?id=25543859 https://news.ycombinator.com/item?id=25424852

Wasn't aware of that.

Resubmitting interesting content that hasn't got traction earlier on is however explicitly allowed in the guidelines IIRC.

Re: Only Google is really allowed to crawl the web

#102

Post by bombcar »

slymon99 wrote: post #88
danielscrubs wrote: post #80

Earlier quoted context omitted.

Be careful when using Google Flight, last time I checked they use significantly less margins between flights so it’s shorter trips but much riskier.

Can you elaborate on this? Do you mean shorter layovers?

It sounds like it - and third-party companies will often show you flights that involve different companies on the different legs - which can leave you in a pickle because technically each airline's job is to get you to the end of THIER flight, not the entire journey.

Re: Only Google is really allowed to crawl the web

#103

Post by quantumofalpha »

tomc1985 wrote: post #91

I think the solution here is everybody masquerades as Googlebot so we can render the whole thing moot

Ignoring robots.txt is trivial, that's why some(many?) sites enforce it by verifying source IP and recognize Googlebot from its IP addresses - how will you get access to one of those?

Re: Only Google is really allowed to crawl the web

#104

Post by systemBuilder »

This is not really about Google.

Websites block crawlers because they get abused / crashed by Crawlers. In the early days (2000-2010) Google not only got banned by some websites, it even got DNS-banned for abusing some DNS domains. You see, Google already has already built the "megacrawlers" described in this article, it can melt any website on the Internet, even Facebook - the largest, and they paid a high price for letting the early Google crawlers run free.

Google today has a rate-limit for every single website and DNS sub-domain on the internet. For small websites the default is a handful of web pages every few seconds. Google has a very slow (days) algorithm to increase its crawl rate, and a very fast (1d) algorithm to cut the rate limit if it's getting any of the errors likely due to website overload.

To summarize, Google has several layers of congestion control custom-designed into the crawl application. Most small web crawlers have zero.

None of these other crawlers have figured this out, so they abuse websites, causing all small-scale crawlers to get banned.

- ex-Google Crawl SRE

Re: Only Google is really allowed to crawl the web

#105

Post by adolph »

sxp wrote: post #3

https://knuckleheads.club/the-googlebot-monopoly/ has actual details. > Let’s take a look at the robots.txt for census.gov from October of 2018 as a specific example to see how robots.txt files typically work. This document is a good example of a common pattern. The first two lines of the file specify that you cannot crawl census.gov unless given explicit permission. The rest of the file specifies that Google, Micros…

Is it legal for a government entity to issue a robots.txt like that? Maybe the line between use and abuse hasn't been delinated as well as it needs to be.

Re: Only Google is really allowed to crawl the web

#106

Post by foobarian »

magicalist wrote: post #95
dmitriid wrote: post #90

Earlier quoted context omitted.

Google was/is also the largest sponsor of Mozilla. This doesn't stop Google from sabotaging Mozilla. 2 mln is probably Google's hourly profit. For that they get one of the biggest knowledge bases in the world. It's basically free as far as Google is concerned. The instant Google becomes confident they can supplant Wikipedia, they will.

> Google was/is also the largest sponsor of Mozilla. This doesn't stop Google from sabotaging Mozilla. Google isn't a sponsor of Mozilla, they're a customer. Do people think Google is "sponsoring" Apple with $1.5 billion a year too?

> they're a customer.

The cynic in me thinks the product is anti-trust insurance.

Re: Only Google is really allowed to crawl the web

#107

Post by monkeybutton »

jinseokim wrote: post #19

This has been submitted to HN quite a few times. https://news.ycombinator.com/item?id=25426662 (Most comments; 11 comments) https://news.ycombinator.com/item?id=25417067 (3 comments) https://news.ycombinator.com/item?id=25546867 (Most recent; 89 days ago) https://news.ycombinator.com/item?id=25543859 https://news.ycombinator.com/item?id=25424852

Interesting that the most comments it got before was 11, and today it succeeds and makes it to the front page! This is a good illustration of whether or not submissions get any traction can be fairly stochastic.

On topic, stack overflow does exactly what the article is talking about; They lock down their sitemap and make special exceptions for the Google bot:

https://meta.stackexchange.com/a/98087

https://meta.stackexchange.com/questions/33965/how-does-stac...

I can understand SO's reasoning but it only perpetuates the incumbents' stranglehold on the internet.

Re: Only Google is really allowed to crawl the web

#108

Post by tyingq »

magicalist wrote: post #92
tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

> You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Wikipedia visitors, edits, and revenue are all increasing, and the rate that they're increasing is increasing, at least in the last few years. Is this a claim about the third derivative? > Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for. T…

See https://searchengineland.com/wikipedia-confirms-they-are-ste... from 2015. Google's widgets that present Wikipedia data do reduce visitors to Wikipedia.

Or see page views on English Wikipedia from 2016-current: https://stats.wikimedia.org/#/en.wikipedia.org/reading/total... Looks pretty flat, right? Does that seem normal?

As for Wikimedia Enterprise, you do have to read between the lines a bit. "The focus is on organizations that want to repurpose Wikimedia content in other contexts, providing data services at a large scale".

Re: Only Google is really allowed to crawl the web

#109

Post by hannob »

I have seen sites behave differently if you use a Googlebot UA, but am I missing something or does this merely mean that anyone doing something like this

curl -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'

will get Google-level crawler access?

Re: Only Google is really allowed to crawl the web

#110

Post by the_dege »

TheAdamAndChe wrote: post #51
sxp wrote: post #3

https://knuckleheads.club/the-googlebot-monopoly/ has actual details. > Let’s take a look at the robots.txt for census.gov from October of 2018 as a specific example to see how robots.txt files typically work. This document is a good example of a common pattern. The first two lines of the file specify that you cannot crawl census.gov unless given explicit permission. The rest of the file specifies that Google, Micros…

Are there any actual repercussions for just ignoring robots.txt?

Sometimes website admins will also try to report your ips to the service provider as a source of attacks (even if not true).
Post reply on HN