Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only Google is really allowed to crawl the web

knuckleheads.club

Post reply on HN
Print view View original

131–140 of 365 posts

Re: Only Google is really allowed to crawl the web

#131

Post by suicas »

dillondoyle wrote: post #117
suicas wrote: post #18

Earlier quoted context omitted.

A company I worked for ~7 years ago ran its own focused web crawler (fetching ~10-100m pages per month, targeting certain sections of the web). There were a surprising number of sites out there that explicitly blocked access to anyone but Google/Bing at the time. We'd also get a dozen complaints or so a month from sites we'd crawled. Mostly upset about us using up their bandwidth, and telling us that only Google was…

Isn't that the website owners right though? I'm not sure I understand the problem here. If Google is taking traffic and reducing revenue, a company can deny in robots.txt. Google will actually follow those rules - unlike most others that are supposedly in this 2nd class.

Yup, no problem here, was just making an observation about how common such blocking was (and about the fact that some people were upset at being crawled by someone other than Google, despite not blocking them).

The company did respect robots.txt, though it was initially a bit of a struggle to convince certain project managers to do so.

Re: Only Google is really allowed to crawl the web

#132

Post by kirubakaran »

hannob wrote: post #109

I have seen sites behave differently if you use a Googlebot UA, but am I missing something or does this merely mean that anyone doing something like this curl -A 'Mozilla/5.0 (compatible; Googlebot/2.1; + http://www.google.com/bot.html )' will get Google-level crawler access?

That would work on website that have a naive check for just user agent. Google also publishes the IP address ranges their crawlers run on. Lot of websites check for that, and there's no way around that.

https://developers.google.com/search/docs/advanced/crawling/...

Re: Only Google is really allowed to crawl the web

#133

Post by njharman »

JackFr wrote: post #93
sxp wrote: post #3

https://knuckleheads.club/the-googlebot-monopoly/ has actual details. > Let’s take a look at the robots.txt for census.gov from October of 2018 as a specific example to see how robots.txt files typically work. This document is a good example of a common pattern. The first two lines of the file specify that you cannot crawl census.gov unless given explicit permission. The rest of the file specifies that Google, Micros…

So from the website's point of view there is no difference between 'crawling' and 'scraping'. Census.gov I assume has a ton of very useful information which is in the public domain which a host of potential companies could monetize by regularly scraping census.gov. Census.gov's purpose to make this information available to people is served by google, yahoo and bing. On the other hand if I have a business which is bas…

I'm generally anti business. But I have to disagree. "The Public" that the government serves includes businesses. Businesses (ignoring corporate personhood bullshit) are owned and operated by people.

I do not want the government deciding "what purposes" e.g. non-commercial, serve the public good. The public gets to decide that. (charging a license for commercial use is maybe ok (assuming supporting that use costs government "too much").

And I very do not want current situation with the government anointing a handful of corporations (the farthest thing from the public possible) access and denying everyone else including all of the actual public.

Re: Only Google is really allowed to crawl the web

#134

Post by Scoundreller »

bombcar wrote: post #102
slymon99 wrote: post #88

Earlier quoted context omitted.

Can you elaborate on this? Do you mean shorter layovers?

It sounds like it - and third-party companies will often show you flights that involve different companies on the different legs - which can leave you in a pickle because technically each airline's job is to get you to the end of THIER flight, not the entire journey.

And sometimes with a change of airport!

Re: Only Google is really allowed to crawl the web

#136

Post by tyingq »

SamBam wrote: post #129
tyingq wrote: post #108

Earlier quoted context omitted.

See https://searchengineland.com/wikipedia-confirms-they-are-ste... from 2015. Google's widgets that present Wikipedia data do reduce visitors to Wikipedia. Or see page views on English Wikipedia from 2016-current: https://stats.wikimedia.org/#/en.wikipedia.org/reading/total... Looks pretty flat, right? Does that seem normal? As for Wikimedia Enterprise, you do have to read between the lines a bit. "The focus is on o…

The first link doesn't seem quite conclusive (see the part at the bottom), and also doesn't give evidence that Google's widgets are to blame. The flattening of users could also be due to a general internet-wide reduction in long-form (or even medium-form) non-fiction reading. How are page views for The New York Times? Seems like it should be simple to A/B test, though. Obviously Google could do it themselves by rando…

Edit: Removed bad "simple english graph", thanks. Though the regular english wikipedia traffic is flat from 2016-present.

As for NYT, is there a better proxy to compare to? There's no public pageview stats and they have a paywall.

Re: Only Google is really allowed to crawl the web

#138

Post by dheera »

soheil wrote: post #17
henriquez wrote: post #2

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to re…

It's hilarious to think there exists people who think googlebot does not get special treatment from website operators. Here is an experiment you can do in a jiffy, write a script that crawls any major website and see how many URL fetches it takes before your IP gets blocked. Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.

Do any of them intersect with Google Cloud IP addresses? If so set up a VPN server on Google Cloud.

Re: Only Google is really allowed to crawl the web

#139

Post by jszymborski »

monkeybutton wrote: post #107
jinseokim wrote: post #19

This has been submitted to HN quite a few times. https://news.ycombinator.com/item?id=25426662 (Most comments; 11 comments) https://news.ycombinator.com/item?id=25417067 (3 comments) https://news.ycombinator.com/item?id=25546867 (Most recent; 89 days ago) https://news.ycombinator.com/item?id=25543859 https://news.ycombinator.com/item?id=25424852

Interesting that the most comments it got before was 11, and today it succeeds and makes it to the front page! This is a good illustration of whether or not submissions get any traction can be fairly stochastic. On topic, stack overflow does exactly what the article is talking about; They lock down their sitemap and make special exceptions for the Google bot: https://meta.stackexchange.com/a/98087 https://meta.stacke…

I think it's partly because they create a website which reported on the status of the Ever Given which rose to 1. on the front page.

I feel like I often see submissions which are, even tangentially, related to front page material rise very quickly.

Regardless, congrats to Knuckleheads Club for fighting the good fight.

Re: Only Google is really allowed to crawl the web

#140

Post by SV_BubbleTime »

danielscrubs wrote: post #80
throwaway_kufu wrote: post #27

Earlier quoted context omitted.

They are not just taking away internet traffic, but in the flights example, they actually acquired an aggregate flight/travel company and so they are actually entering markets and competing with their own ad customers. Then it comes fully circle to Google unfairly using their market position vis-a-vis data, search and advertising. It’s a win-win Google lets the data dictate which markets to enter and on one hand they…

Be careful when using Google Flight, last time I checked they use significantly less margins between flights so it’s shorter trips but much riskier.

Oh don’t worry, you have 15 on-paper minutes to go from A1 to A70 in Detroit... in January... and the shuttle is down.
Post reply on HN