Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google’s Web Crawler Bypasses Paywalls

elaineou.com

Post reply on HN
Print view View original

181–190 of 243 posts

Re: How Google’s Web Crawler Bypasses Paywalls

#181

Post by elaineo »

GigabyteCoin wrote: post #132

I was under the impression that the "hack" whereby you searched for the article on Google and clicked through to that article (effectively skipping over the paywall) was a demand of Google's and not an oversight by the paywalled website. I thought that google deemed providing search results which were behind paywalls as a "bad experience" for their search users, and would penalize websites for doing so. Is this no lo…

Google doesn't demand anything. If your paywalled website is not accessible by Google's crawler, then Google will not index it. Publishers want Google to index their pages and drive potential paying visitors, which is why they open the loophole themselves.

For the second point, Google does require that publishers specify "registration required" in their sitemap.

Re: How Google’s Web Crawler Bypasses Paywalls

#182

Post by ikeboy »

LunaSea wrote: post #177
ikeboy wrote: post #55

New workaround: paste the article title into archive.is. I don't know what they're doing but they have a workaround of some sort.

That's actually really interesting! Could anyone chime in and explain how they might work around this issue?

My guess is they have a login, but they could just be using some workaround like described in OP.

I've used them to save Facebook posts before, and the pages were logged in to some "Nathan" IIRC. They probably have a bunch of hacks for specific sites that needed fixing.

Re: How Google’s Web Crawler Bypasses Paywalls

#183

Post by morgante »

eps wrote: post #28

Correct me if I'm wrong, but wasn't there a long standing Google's policy that the version of the page served to their crawler must also be publicly accessible. That would then be the reason why WSJ articles were accessible through the paste-into-google trick, rather than because WSJ was incompetent and failed to "fix" the bypass. So does it mean that Google will no longer index full WSJ articles or does it mean a ch…

You are correct, Google requires that you let users see the first click for free if you want to index content behind a paywall. [1]

Since this is billed as an "experiment" I'm guessing that WSJ is just testing the waters. If they roll it out to everyone, they will have to serve only snippets to Google or risk getting delisted.

[1] https://support.google.com/news/publisher/answer/40543?hl=en

Re: How Google’s Web Crawler Bypasses Paywalls

#184

Post by morgante »

pacquiao882 wrote: post #113
elaineo wrote: post #9

Earlier quoted context omitted.

Google is not okay with cloaking, but they will whitelist publishers if the publisher specifically includes a parameter that declares if the site requires registration or subscription. This is done in the sitemap. https://support.google.com/news/publisher/answer/74288?hl=en

WSJ is big enough to negotiate their own terms with Google Search.

They're really not. They need Google a lot more than Google needs them.

One of these things is going to happen:

(1) They end this "experiment."

(2) They stop serving Google the full content. (And see their rankings drop accordingly.)

(3) They get delisted for cloaking.

Re: How Google’s Web Crawler Bypasses Paywalls

#185

Post by jd3 »

jrockway wrote: post #45
zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

User agent strings have a long history of being intentionally misleading. IE 11 claims to be "Mozilla/5.0". Chrome claims to be "Safari/537.36". The User-Agent string is all lies, and has been ever since the first site started doing UA sniffing.

I couldn't believe my eyes when I saw that netflix started sniffing the UA. You can't even play a video in SeaMonkey 2.39 without the incredibly stupid

general.useragent.override.netflix.com;Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:42.0) Gecko/20100101 Firefox/42.0

in about:config... what were they thinking?

Re: How Google’s Web Crawler Bypasses Paywalls

#186

Post by melted »

lloyddobbler wrote: post #68

"Remember: Any time you introduce an access point for a trusted third party, you inevitably end up allowing access to anybody." See also: http://www.apple.com/customer-letter/ :)

Except of course what FBI is proposing wouldn't give access to "anybody". Just unfettered access for FBI, CIA and NSA by way of gag orders and national security letters, forcing Apple to break security of their hardware and not speak about it publicly. Of note is the fact that it wouldn't even give direct access to FBI, since they don't have the firmware keys (yet).

Re: How Google’s Web Crawler Bypasses Paywalls

#187

Post by Implicated »

CM30 wrote: post #117
slig wrote: post #7

If they're now blocking clicks from Google, doesn't that mean that they're cloaking and violating the Google's Webmaster Guidelines [1]? [1]: https://support.google.com/webmasters/answer/66355?hl=en

Yes, and I suspect that's what going to get them to change it back again. After a ban from Google and the traffic drop it'll likely bring, that paywall is likely coming right back down. An awful lot of media companies made similar mistakes before, and it's always ended with them quickly removing their 'work arounds'.

Yea, except they're not going to get that ban.

Re: How Google’s Web Crawler Bypasses Paywalls

#188

Post by [deleted] »

oneeyedpigeon wrote: post #152
swanson wrote: post #40

Earlier quoted context omitted.

I wonder if a user-agent that was something like "Not a Googlebot" would a) allow access (probably regex based) and b) be truthful/plausible deniability.

What is your purpose in setting your user-agent string to that value, other than cleverly bypassing the paywall?

[deleted]

Re: How Google’s Web Crawler Bypasses Paywalls

#189

Post by ivansavz »

zem wrote: post #35

i thought of doing that when the "search google" trick stopped working, but i decided it crossed the point where i would feel like i was unfairly circumventing their clear desire not to serve me the content. i've just added wsj to my mental ignore list and count it as a few more minutes gained to do something else.

Yeah, same here. Every time I get to a tab where I see a paywall I just close that tab, probably saving 5-10 mins of my life!

Re: How Google’s Web Crawler Bypasses Paywalls

#190

Post by kenshaw »

Basically, the article is stating to change the User-Agent to GoogleBot or Bing or whatever other crawler UA you'd prefer. While that's doable, that's something that is easily detectable and prevented, as all of the big crawlers can be validated against DNS.

Additionally, I would like to point out that I wrote a Varnish extension for the express purpose of validating User-Agent strings through DNS lookups, and is available here: https://github.com/knq/libvmod-dns

It was built because we had specifically a problem with bad bots crawling a large site (multiply.com) and this was one of the easiest ways to filter out the bad bots from the good, and to enforce robots.txt policies on a per bot basis. It works very well, as you can do any kind of DNS caching internally and prevent this kind of behavior, if that's your goal.

Post reply on HN