Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only Google is really allowed to crawl the web

knuckleheads.club

Post reply on HN
Print view View original

151–160 of 365 posts

Re: Only Google is really allowed to crawl the web

#151

Post by judge2020 »

simias wrote: post #31
zmarty wrote: post #6

Earlier quoted context omitted.

A lot of news websites restrict any crawler other than Google. And this does not happen only via robots.txt.

Indeed, years ago I had scripts to automatically fetch URLs from IRC and I quickly realized that if I didn't spoof the user agent of a proper web browser many websites would reject the query. Googlebot's UA worked just fine however.

> Googlebot's UA worked just fine however

They obviously don't care enough then - Google says you should use rdns to verify that googlebot crawls are real[0]. Cloudflare does this automatically now as well for customers with WAF (pro plan).

0: https://developers.google.com/search/docs/advanced/crawling/...

Re: Only Google is really allowed to crawl the web

#152

Post by jedberg »

dmitriid wrote: post #90
onlyrealcuzzo wrote: post #71

Earlier quoted context omitted.

Wikipedia isn't monetized. Doesn't it benefit them if Google is serving their content for free and people are finding the information they want without having to hit Wikipedia?? And also, isn't Google the largest sponsor for Wikipedia already? In 2019 - Google donated $2M [1]. In 2010, Google also donated $2m [2]. [1] https://techcrunch.com/2019/01/22/google-org-donates-2-milli... [2] https://en.wikipedia.org/wiki/Wi…

Google was/is also the largest sponsor of Mozilla. This doesn't stop Google from sabotaging Mozilla. 2 mln is probably Google's hourly profit. For that they get one of the biggest knowledge bases in the world. It's basically free as far as Google is concerned. The instant Google becomes confident they can supplant Wikipedia, they will.

> 2 mln is probably Google's hourly profit.

You don't have to guess, their numbers are public. In 2020 they made $40B in profit, so it takes them about 27 minutes to make $2M in profit.

Re: Only Google is really allowed to crawl the web

#153

Post by judge2020 »

monkeybutton wrote: post #107
jinseokim wrote: post #19

This has been submitted to HN quite a few times. https://news.ycombinator.com/item?id=25426662 (Most comments; 11 comments) https://news.ycombinator.com/item?id=25417067 (3 comments) https://news.ycombinator.com/item?id=25546867 (Most recent; 89 days ago) https://news.ycombinator.com/item?id=25543859 https://news.ycombinator.com/item?id=25424852

Interesting that the most comments it got before was 11, and today it succeeds and makes it to the front page! This is a good illustration of whether or not submissions get any traction can be fairly stochastic. On topic, stack overflow does exactly what the article is talking about; They lock down their sitemap and make special exceptions for the Google bot: https://meta.stackexchange.com/a/98087 https://meta.stacke…

> They lock down their sitemap and make special exceptions for the Google bot:

Their robots.txt, on the other hand, is more restrictive of Googlebot:

https://stackoverflow.com/robots.txt

  User-agent: Googlebot-Image
  Disallow: /*/ivc/*
  Disallow: /users/flair/
  Disallow: /jobs/n/*
  ..

Re: Only Google is really allowed to crawl the web

#154

Post by p-sharma »

quantumofalpha wrote: post #103
tomc1985 wrote: post #91

I think the solution here is everybody masquerades as Googlebot so we can render the whole thing moot

Ignoring robots.txt is trivial, that's why some(many?) sites enforce it by verifying source IP and recognize Googlebot from its IP addresses - how will you get access to one of those?

What does "recognize Googlebot from its IP addresses" mean? If I'm a human and I access a site, I have some other IP than Googlebot, how should this side know if I'm a human or knuckleheadsbot?

Re: Only Google is really allowed to crawl the web

#155

Post by treis »

supernovae wrote: post #125
treis wrote: post #50

Earlier quoted context omitted.

>However, that sort of search used to (most times) lead to a visit to the airline web site. I don't think that's correct. In the old days you'd either call a travel agent or use an aggregator like expedia. Google muscles out intermediaries like Expedia, Yelp, and so on. It's not likely much better or worse for the end user or supplier. Just swapping one middleman for another.

I don't think google muscling out intermediaries like Expedia is a good thing. Just for example, Expedia is probably 5% of Google's total revenue and Google doesn't like slim margin services by and large that can't be automated. Travel is fairly high-touch - people centric. It doesn't fit Google's "MO". But... its shitty that google can play all sides of the markets while holding people ransom to mass sums of money t…

>In essence, you're advocating that eBay goes away because google could do it... they could.. and eBay is technically just an intermediary, but do we want everything to be googlefied?

I don't think I'm really advocating for it as much as I see as a more or less neutral change.

That said, I'm pretty ambivalent about Google. Their size is a concern, but they also tend to be pretty low on the dark pattern nonsense. eBay, to use an example you gave, screwed me out of some buyer protection because of poor UX and/or bug (I never saw the option to claim my money after the seller didn't respond). In this specific instance Google ends the process by sending you to the airline to complete the booking. That, imho, is likely better than dealing with Expedia.

Re: Only Google is really allowed to crawl the web

#156

Post by pessimizer »

njharman wrote: post #133
JackFr wrote: post #93

Earlier quoted context omitted.

So from the website's point of view there is no difference between 'crawling' and 'scraping'. Census.gov I assume has a ton of very useful information which is in the public domain which a host of potential companies could monetize by regularly scraping census.gov. Census.gov's purpose to make this information available to people is served by google, yahoo and bing. On the other hand if I have a business which is bas…

I'm generally anti business. But I have to disagree. "The Public" that the government serves includes businesses. Businesses (ignoring corporate personhood bullshit) are owned and operated by people. I do not want the government deciding "what purposes" e.g. non-commercial, serve the public good. The public gets to decide that. (charging a license for commercial use is maybe ok (assuming supporting that use costs gov…

A specific case where this favorite-picking by government enables corruption: https://en.wikipedia.org/wiki/Nationally_recognized_statisti...

And an example from the quickly-approaching future, when there will be Nationally Recognized Media Organizations who license "Fact-Checkers," through which posts to public-facing will have to be submitted for certification and correction.

Re: Only Google is really allowed to crawl the web

#157

Post by tedunangst »

soheil wrote: post #17
henriquez wrote: post #2

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to re…

It's hilarious to think there exists people who think googlebot does not get special treatment from website operators. Here is an experiment you can do in a jiffy, write a script that crawls any major website and see how many URL fetches it takes before your IP gets blocked. Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.

I don't whitelist googlebot, but I don't block them either because their crawler is fairly slow and unobtrusive. Other crawlers seem determined to download the entire site in 60 seconds, and then download it again, and again, until they get banned.

Re: Only Google is really allowed to crawl the web

#158

Post by noxvilleza »

henriquez wrote: post #2

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to re…

Google aren't the bad actor in the sense that they are actively doing something wrong, but they are definitely benefiting from the monopoly that they created and work on maintaining. If this continues then nobody will really ever be able to challenge them, which means possibly "better" products will fail to penetrate the market.

Re: Only Google is really allowed to crawl the web

#159

Post by judge2020 »

p-sharma wrote: post #124

Maybe a naïve question but what prevents Knuckleheads’ from ignoring the robots.txt and crawl the side anyway? And if it's so easy to do, how does Google have a monopoly on crawling then?

It's just rude to do so, and there are some technical issues with doing that as well (such as crawling admin panel which might trigger backend alarms/security alerts). Google also doesn't have a legal monopoly on crawling, only a natural monopoly thanks to a lot of websites independently choose to only allow Google and Bing because of the many issues with third-party crawlers (eg. crawling all pages at once, costing money/slowing down the site[0]).

0: https://news.ycombinator.com/item?id=26593722

Re: Only Google is really allowed to crawl the web

#160

Post by Spivak »

supert56 wrote: post #116
tyingq wrote: post #12

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google prese…

Perhaps I am misunderstanding or over simplifying things but it always surprises me that there are legal cases brought against companies who scrape data when so many of Google's products are doing exactly this. It definitely feels like one set of rules for them and a different set for everyone else.

I mean it's not that weird that a company would authorize major search engines scraping them but no one else.

I don't really see this as Google playing by different rules so much as economic incentives being aligned in Google's favor.

Post reply on HN