How Google’s Web Crawler Bypasses Paywalls • Hacker News Board

Re: How Google’s Web Crawler Bypasses Paywalls

#131

Post by hughes » Fri, Feb 19, 2016, 9:39 PM UTC

zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

Is that significantly different from viewing google's cached result of their legitimate access? The result is the same.

Re: How Google’s Web Crawler Bypasses Paywalls

#132

Post by GigabyteCoin » Fri, Feb 19, 2016, 9:44 PM UTC

I was under the impression that the "hack" whereby you searched for the article on Google and clicked through to that article (effectively skipping over the paywall) was a demand of Google's and not an oversight by the paywalled website.

I thought that google deemed providing search results which were behind paywalls as a "bad experience" for their search users, and would penalize websites for doing so.

Is this no longer the case?

Re: How Google’s Web Crawler Bypasses Paywalls

#133

Post by fiatmoney » Fri, Feb 19, 2016, 9:54 PM UTC

zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

If I run a car wash with a sign, "red cars washed free", and you paint your car red, have you defrauded me? You're not really a red car, you're just pretending!

Re: How Google’s Web Crawler Bypasses Paywalls

#134

Post by jsprogrammer » Fri, Feb 19, 2016, 9:55 PM UTC

Spare_account wrote: post #98

oldmanjay wrote: post #91

Earlier quoted context omitted.

As you've indicated, a request can only be considered malformed if it doesn't conform to the standards. Here is what the relevant RFC[0] has to say about the User-Agent header: > Likewise, implementations are encouraged not to use the product tokens of other implementations in order to declare compatibility with them, as this circumvents the purpose of the field. If a user agent masquerades as a different user agent,…

Unless you were being tongue-in-cheek, you've completely sidestepped the intent of my comment and continued with the linguistic silliness. Obtaining paywall-protected content by faking your user agent to purport yourself to be a Google Crawler is quite clearly fraudulent. This isn't a point for debate. PS. To play along with the linguistic theme, can you provide a source for the definition of a malformed request? My…

So...if I only entitle Mozilla based agents to view my content and Chrome, IE, and Safari users still get access by claiming to be a Mozilla agent, those users have committed a felony?

This line of argumentation is beyond ridiculous.

Re: How Google’s Web Crawler Bypasses Paywalls

#135

Post by wmeredith » Fri, Feb 19, 2016, 9:57 PM UTC

fiatmoney wrote: post #133

zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

If I run a car wash with a sign, "red cars washed free", and you paint your car red, have you defrauded me? You're not really a red car, you're just pretending !

Oh pish posh. If you forge a key that fits the lock on my front door, yeah, your breaking the law. The lock itself is not an advertisement to come in if you can unlock it.

Re: How Google’s Web Crawler Bypasses Paywalls

#136

Post by tete » Fri, Feb 19, 2016, 10:00 PM UTC

Doesn't Google usually try to punish websites that show users something different and even mentions that somewhere?

Not an SEO Expert here, but wonder how and whether Google will end up handling that. I mean making an exception could also be considered abuse of power in some countries of the world. Don't have any strong opinion yet on that, just saying that because of how the EU exercised certain laws in recent years.

Re: How Google’s Web Crawler Bypasses Paywalls

#137

Post by TheDong » Fri, Feb 19, 2016, 10:01 PM UTC

mcherm wrote: post #119

oldmanjay wrote: post #91

Earlier quoted context omitted.

As you've indicated, a request can only be considered malformed if it doesn't conform to the standards. Here is what the relevant RFC[0] has to say about the User-Agent header: > Likewise, implementations are encouraged not to use the product tokens of other implementations in order to declare compatibility with them, as this circumvents the purpose of the field. If a user agent masquerades as a different user agent,…

> a request can only be considered malformed if it doesn't conform to the standards That is demonstrably false. I, personally, can consider any request malformed unless it starts with the letter W. Regardless of what the standards say, I can think whatever I want to. Similarly, the law can make up whatever rules IT wants to about the definition of "malformed". In that case, it pays some scant attention to things like…

That argument doesn't matter thought. He didn't say "a request can only be considered malformed by mcherm's definition of the term if...".

I could say "the sky is blue" and you could say "demonstrably false; the sky is personally red because I'm wearing colour-altering sunglasses".

What he meant is obviously "a request can only be considered malformed (as defined by the commonly accepted definition of malformed) .."

Your argument is fallacious. The courts will use common definitions of terms and have regard for context.

However, what you're arguing about is even more pointless. The CFAA doesn't depend on the term "malformed" in any way, but on the term "authorized access".

Your first sentence is not similar at all to your second; they're demonstrating first a blatant disregard for common definitions of terms, followed by praise of such common definitions.

Re: How Google’s Web Crawler Bypasses Paywalls

#138

Post by tomcam » Fri, Feb 19, 2016, 10:03 PM UTC

sneak wrote: post #123

CyberDildonics wrote: post #109

Earlier quoted context omitted.

Unauthorized? If you put up a sign and tell people they have to pay to look at it, is it illegal to look at it and not pay? This should be a rhetorical question.

[I am not a lawyer and this is not legal advice.] It is not specifically defined in the law, so it reverts to the traditional meaning: anything the owner of the system says you aren't authorized to access. It's lunacy, I know. That's what HTTP headers and WAFs and such are for. But that's the stupid law, and it sent someone who used to be my friend to federal prison for changing a user agent and referrer and accessin…

Whoa. Care to tell that story or link to it?

Re: How Google’s Web Crawler Bypasses Paywalls

#139

Post by fiatmoney » Fri, Feb 19, 2016, 10:06 PM UTC

wmeredith wrote: post #135

fiatmoney wrote: post #133

Earlier quoted context omitted.

If I run a car wash with a sign, "red cars washed free", and you paint your car red, have you defrauded me? You're not really a red car, you're just pretending !

Oh pish posh. If you forge a key that fits the lock on my front door, yeah, your breaking the law. The lock itself is not an advertisement to come in if you can unlock it.

Stop it with the tired lock metaphors, they are wildly inapplicable. If the WSJ wanted to require access to some secret knowledge or possession of some token for access, they could. Instead, they are dispatching on something the protocol they're using agrees you are free to set to whatever you want for the "best experience". In fact the specification specifically recognizes that you will change the agent. Much like the color of your car.

"If a user agent masquerades as a different user agent, recipients can assume that the user intentionally desires to see responses tailored for that identified user agent, even if they might not work as well for the actual user agent being used"

It's a param exactly equivalent to asking "who would you like to be treated as?" If I say "the pope!" and you respond by kissing my ring, I have not defrauded you.

Re: How Google’s Web Crawler Bypasses Paywalls

#140

Post by TheDong » Fri, Feb 19, 2016, 10:07 PM UTC

fiatmoney wrote: post #133

zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

If I run a car wash with a sign, "red cars washed free", and you paint your car red, have you defrauded me? You're not really a red car, you're just pretending !

Terrible analogy.

You're talking about a sign with practically no legal meaning.

This discussion is about a law which does have legal meaning.

To fix your analogy, it would be "If there's a law that says only explicitly authorized cars are allowed in a car wash; else 5 years in prison, and I have a car wash and say 'only red cars' ...". Of course, that still doesn't properly capture this since the intent of the law also matters and that law would be senseless, and so interpretation would be less obvious.

The better analogy is the one below about breaking into homes being illegal, but what if you happen to have a key that fits the lock? (though that also is a bad analogy in its own way).

Basically, making physical analogies for technical matters is rarely correct. It's often the best way to convince non-technical people of a matter without them needing to actually understand it.