Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google’s Web Crawler Bypasses Paywalls

elaineou.com

Post reply on HN
Print view View original

11–20 of 243 posts

Re: How Google’s Web Crawler Bypasses Paywalls

#11

Post by slig »

elaineo wrote: post #9
slig wrote: post #7

If they're now blocking clicks from Google, doesn't that mean that they're cloaking and violating the Google's Webmaster Guidelines [1]? [1]: https://support.google.com/webmasters/answer/66355?hl=en

Google is not okay with cloaking, but they will whitelist publishers if the publisher specifically includes a parameter that declares if the site requires registration or subscription. This is done in the sitemap. https://support.google.com/news/publisher/answer/74288?hl=en

I didn't know that, thanks. But reading, it seems that option is about Google News, not the main Google Search.

> News-specific tag definitions

> Yes, if access is not open, else should be omitted

> Possible values include "Subscription" or "Registration", describing the accessibility of the article. If the article is accessible to Google News readers without a registration or subscription, this tag should be omitted.

Re: How Google’s Web Crawler Bypasses Paywalls

#13

Post by skocznymroczny »

crazysim wrote: post #4

Doesn't this kind of also hurt SEO? I'm would guess Google has some automated system to detect and apply a negative signal to sites that provide different content to a Googlebot user agent than a non-Googlebot user agent. I guess these sites are counting that the other signals outweigh that negative hit. Otherwise, why would expertsexchange be obligated to provide the answers at the very bottom? Did something change?

expert sex change

Re: How Google’s Web Crawler Bypasses Paywalls

#16

Post by lstamour »

mrmcd wrote: post #3
dude_abides wrote: post #2

Or simply use incognito mode and click on Google search result.

Did you read the article. It talks about how that trick no longer works on a lot of sites because they are now checking User-Agent strings too.

Actually I noticed sites have simply changed policies -- if you're a regular visitor your cookies will identify you and block content. The Incognito mode trick works for WSJ and others that would still check the referrer header. Allowing Googlebot access and checking the referrer header are two different things.

Also, Google has published IP addresses it uses, so this extension might not last long...

Re: How Google’s Web Crawler Bypasses Paywalls

#17

Post by jgh »

I just tried clicking on "Harper Lee, Author of ‘To Kill a Mockingbird,’ Dies at Age 89" from wsj.com's homepage and got the paywall.

I then pasted the headline into google and clicked on it from Google results and did not get hit by the paywall.

Re: How Google’s Web Crawler Bypasses Paywalls

#18

Post by e40 »

jgh wrote: post #17

I just tried clicking on "Harper Lee, Author of ‘To Kill a Mockingbird,’ Dies at Age 89" from wsj.com's homepage and got the paywall. I then pasted the headline into google and clicked on it from Google results and did not get hit by the paywall.

Mine did, which surprised me, and I did exactly the same thing.

Re: How Google’s Web Crawler Bypasses Paywalls

#19

Post by dogma1138 »

sylvinus wrote: post #8

Well, that trick won't last long either. It's trivial to verify that an IP indeed belongs to Google: https://support.google.com/webmasters/answer/80553?hl=en

Seems to work if you deploy a proxy on Google's app engine and use it to access WSJ ;)

Re: How Google’s Web Crawler Bypasses Paywalls

#20

Post by slig »

dogma1138 wrote: post #19
sylvinus wrote: post #8

Well, that trick won't last long either. It's trivial to verify that an IP indeed belongs to Google: https://support.google.com/webmasters/answer/80553?hl=en

Seems to work if you deploy a proxy on Google's app engine and use it to access WSJ ;)

A App Engine wouldn't have a IP with a reverse DNS *.googlebot.com, would it?
Post reply on HN