Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google’s Web Crawler Bypasses Paywalls

elaineou.com

Post reply on HN
Print view View original

141–150 of 243 posts

Re: How Google’s Web Crawler Bypasses Paywalls

#141

Post by TheDong »

hughes wrote: post #131
zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

Is that significantly different from viewing google's cached result of their legitimate access? The result is the same.

The difference is intent!

The law says "authorized access" and the WSJ authorized Google to access their content in order to index it. The WSJ did not authorize that content to be presented, for free, to you, an end user, necessarily. I don't know which way a court would rule on it, but it's definitely not black-and-white.

Sure, it's technically similar, but the court doesn't care. The court doesn't care if the law makes no technical sense, because it's a law, not a program.

Re: How Google’s Web Crawler Bypasses Paywalls

#142

Post by djsumdog »

comex wrote: post #116
lojack wrote: post #103

Earlier quoted context omitted.

In my opinion the WSJ has a major rendering bug that affects everyone except for the google search crawler, and I selectively set my User Agent to get around said bug.

In any judge's opinion, you'd be full of shit. Judges aren't stupid. Which doesn't matter, because the WSJ is never going to sue you. But make sure you only consider your justification a personal one, not one that would provide any legal protection.

Most judges and legislators are fucking morons. Legal systems aren't a science. I wrote a thing on it a few years ago:

http://khanism.org/security/legality/

Re: How Google’s Web Crawler Bypasses Paywalls

#143

Post by Spare_account »

jsprogrammer wrote: post #134
Spare_account wrote: post #98

Earlier quoted context omitted.

Unless you were being tongue-in-cheek, you've completely sidestepped the intent of my comment and continued with the linguistic silliness. Obtaining paywall-protected content by faking your user agent to purport yourself to be a Google Crawler is quite clearly fraudulent. This isn't a point for debate. PS. To play along with the linguistic theme, can you provide a source for the definition of a malformed request? My…

So...if I only entitle Mozilla based agents to view my content and Chrome, IE, and Safari users still get access by claiming to be a Mozilla agent, those users have committed a felony? This line of argumentation is beyond ridiculous.

Are your Mozilla users paying for access? Are the Chrome users with the fake agent strings obtaining paid-for content for free? The use of forged agent strings isn't the offence.

Re: How Google’s Web Crawler Bypasses Paywalls

#144

Post by mrgrieves »

elaineo wrote: post #86
mattmaroon wrote: post #79

Earlier quoted context omitted.

Yep. I then pasted it but it didn't work on wsj.com. Oh well.

try deleting cookies, then hit refresh.

Not working here either. I tried wiping out cookies and cache, and Chrome is sending the right user agent (Googlebot).

Re: How Google’s Web Crawler Bypasses Paywalls

#145

Post by fiatmoney »

TheDong wrote: post #140
fiatmoney wrote: post #133

Earlier quoted context omitted.

If I run a car wash with a sign, "red cars washed free", and you paint your car red, have you defrauded me? You're not really a red car, you're just pretending !

Terrible analogy. You're talking about a sign with practically no legal meaning. This discussion is about a law which does have legal meaning. To fix your analogy, it would be "If there's a law that says only explicitly authorized cars are allowed in a car wash; else 5 years in prison, and I have a car wash and say 'only red cars' ...". Of course, that still doesn't properly capture this since the intent of the law a…

Fraud statutes do exist and the CFAA is in direct correspondence with them.

If there's anything the tech sector should be able to come to agreement on, it's that lock metaphors in a situation with absolutely no hidden knowledge or private tokens make us all dumber.

Re: How Google’s Web Crawler Bypasses Paywalls

#146

Post by rhino369 »

metafunctor wrote: post #41

I'm pretty sure Google will soon stop indexing WSJ. Why index something if the vast majority of users cannot access the pages behind the links? EDIT: The "paste a headline into Google" trick still works for me, though. If this continues to be the case, they will keep indexing, of course.

>Why index something if the vast majority of users cannot access the pages behind the links?

So people can find it? I'd be pissed if Google de-indexed something like IEEE because it has a paywall.

Assuming the internet has to be freely available is a mistake. Especially with the continued growth of an adblocked internet. We could be facing an internet with significant paywalls in the future.

I'd support a "free" search term to weed out paywalled results.

Furthermore, Google shouldn't be making normative judgements about what people should see. It's an abuse of their monopoly.

Re: How Google’s Web Crawler Bypasses Paywalls

#147

Post by mikedmiked »

ikeboy wrote: post #55

New workaround: paste the article title into archive.is. I don't know what they're doing but they have a workaround of some sort.

This is what I do.

I actually made a bookmarklet with the following pasted into the URL, so you can do it in a single click:

javascript:void(open('https://archive.is/?run=1&url='+encodeURIComponent(document....)

Re: How Google’s Web Crawler Bypasses Paywalls

#148

Post by RIMR »

mbroshi wrote: post #24

Am I alone in feeling like this is akin to a tutorial on how you can shoplift without getting caught? WSJ, for better or worse, does not want to give you content without your paying for it. If you take that content without paying, you are stealing. Just because you have figured out how to get past their security does not mean it's not stealing. (See the second precept here: https://en.wikipedia.org/wiki/Five_Precepts…

> abstain from taking what is not given

I'm not taking, I am simply absorbing information. That information will still be there when I am done reading it. Have I really stolen, or did I just refuse to give someone money on demand?

Re: How Google’s Web Crawler Bypasses Paywalls

#149

Post by eli »

chinathrow wrote: post #56
eli wrote: post #10

Earlier quoted context omitted.

I'm 99% sure I've encountered a Googlebot crawling pages with the UA of a regular browser, presumably for exactly this purpose.

They do this via an iPhone like UA, though googlebot is still in there too.

I'm pretty sure that's just to see if the site is serving different content to mobile vs desktop. I think they also sometimes hit pages with no mention of googlebot.

Re: How Google’s Web Crawler Bypasses Paywalls

#150

Post by effie »

nkrisc wrote: post #129
effie wrote: post #125

Earlier quoted context omitted.

The difference is substantial; in principle, one can modify his request to intentionally bypass the authorization mechanism occurring on the server. One cannot mislead anyone/anything by displaying his data in a customized way in private on his computer.

Fair point. Intent matters. However must it be demonstrated that there is some specific malicious intent behind changing a request header versus simply changing your user agent for the heck of it? In a related point, some news sites load a modal and prevent scrolling over an article asking you to sign up. However if the full article is included in the response and I read it by simply viewing the response body (HTML)…

> must it be demonstrated that there is some specific malicious intent...

I think so. Just the act of changing User-Agent alone does not mean a fraud is happening. User-Agents get changed often for valid reasons - research, detection of cloaking, testing etc.

> However if the full article is included in the response and I read it by simply viewing the response body (HTML) is that circumventing security?

That is a hard issue. I think the answer should be no, it is not circumvention, at least not in the above sense.(\) If you can get full article on your computer, directly readable without fraudulent behaviour on your part, this means that the sender did not place a security measure to guard it, so there can be no circumventing it. Switching javascript off or displaying source text of the document is fully in the control of the requestor and the sender should know it.

(\) Unfortunately, it seems legal systems allow companies to restrict what you do with things they produce, even if you do it on your computer in private, so legally, this may not be successful in court.

Post reply on HN