Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google’s Web Crawler Bypasses Paywalls

elaineou.com

Post reply on HN
Print view View original

91–100 of 243 posts

Re: How Google’s Web Crawler Bypasses Paywalls

#91

Post by oldmanjay »

Spare_account wrote: post #85
jsprogrammer wrote: post #77

Earlier quoted context omitted.

If the computer on the public Internet responds to a standard HTTP request, it has explicitly authorized my access to whatever information it sent me.

Your request wouldn't be 'standard', it would be deliberately malformed to bypass a paywall. You guys are performing linguistic gymnastics to get around that fact.

As you've indicated, a request can only be considered malformed if it doesn't conform to the standards. Here is what the relevant RFC[0] has to say about the User-Agent header:

> Likewise, implementations are encouraged not to use the product tokens of other implementations in order to declare compatibility with them, as this circumvents the purpose of the field. If a user agent masquerades as a different user agent, recipients can assume that the user intentionally desires to see responses tailored for that identified user agent, even if they might not work as well for the actual user agent being used.

That sure sounds like impersonating other user agents is allowed, but not encouraged. That is a clear distinction from being malformed.

[0] https://tools.ietf.org/html/rfc7231#section-5.5.3

Re: How Google’s Web Crawler Bypasses Paywalls

#92

Post by njharman »

mbroshi wrote: post #24

Am I alone in feeling like this is akin to a tutorial on how you can shoplift without getting caught? WSJ, for better or worse, does not want to give you content without your paying for it. If you take that content without paying, you are stealing. Just because you have figured out how to get past their security does not mean it's not stealing. (See the second precept here: https://en.wikipedia.org/wiki/Five_Precepts…

No content is being given or taken. This is restriction on distribution / copying. Massively different ethics than stealing despite what the Copyright Cabal wants you to think.

The only loss is the energy/bandwidth/cycles WSJ servers spent answering your request. Which, I believe, has been basis of computer "fraud" cases.

Re: How Google’s Web Crawler Bypasses Paywalls

#94

Post by AnthonyMouse »

jdunck wrote: post #82

If Google (or any other crawler) wanted to play nice with paywalls, they could issue a public key for their bot, and put a signature in their User Agent string that the domain could then verify. Those signatures could obviously leak, but on a per-domain basis. Perhaps the domains could have a secure way of bumping the valid key generation if they had a leak.

There are two problems with this.

First, they don't want to. In fact, if a search engine can figure out that a link is going to lead to a paywall, they'll probably want to reduce the ranking of the result, because the user is not going to want results they can't actually look at.

Second, it would be a massive antitrust violation because it would prevent access by competing crawlers. The only way around that is to allow access to anyone who claims they're a crawler, which was the original problem.

Re: How Google’s Web Crawler Bypasses Paywalls

#95

Post by compiler-guy »

jsprogrammer wrote: post #77
zaroth wrote: post #76

Earlier quoted context omitted.

Interestingly, the CFAA does not define the term "without authorization" however it does define "exceeding authorization" exactly as I quoted above; - to access a computer with authorization and to use such access to obtain or alter information in the computer that the accesser is not entitled so to obtain or alter. So arguing User-agent is not an authorization mechanism probably won't help you, because exceeding aut…

If the computer on the public Internet responds to a standard HTTP request, it has explicitly authorized my access to whatever information it sent me.

You are confusing an engineering process with a legal one.

Unfortunately, when the prosecutors come, they will only care about the legal process.

Re: How Google’s Web Crawler Bypasses Paywalls

#98

Post by Spare_account »

oldmanjay wrote: post #91
Spare_account wrote: post #85

Earlier quoted context omitted.

Your request wouldn't be 'standard', it would be deliberately malformed to bypass a paywall. You guys are performing linguistic gymnastics to get around that fact.

As you've indicated, a request can only be considered malformed if it doesn't conform to the standards. Here is what the relevant RFC[0] has to say about the User-Agent header: > Likewise, implementations are encouraged not to use the product tokens of other implementations in order to declare compatibility with them, as this circumvents the purpose of the field. If a user agent masquerades as a different user agent,…

Unless you were being tongue-in-cheek, you've completely sidestepped the intent of my comment and continued with the linguistic silliness.

Obtaining paywall-protected content by faking your user agent to purport yourself to be a Google Crawler is quite clearly fraudulent. This isn't a point for debate.

PS. To play along with the linguistic theme, can you provide a source for the definition of a malformed request? My original intent when using the word malformed was not to invoke it's technical definition but rather it's dictionary definition. But, having said that, I just had a 30 second Google hunt and couldn't find anything to corroborate your position.

Re: How Google’s Web Crawler Bypasses Paywalls

#99

Post by hueving »

Based on the comments here, am I to understand that constantly browsing the web with my user agent string set to a googlebot string, I am committing a felony? How would I even know which sites I'm gaining unauthorized access to?

That is completely idiotic if there is a string you can put in a Mozilla browser config that is literally illegal to browse the web with.

Post reply on HN