Live data from Hacker News

How Google’s Web Crawler Bypasses Paywalls

elaineou.com

191–200 of 243 posts

Re: How Google’s Web Crawler Bypasses Paywalls

#191
post #149

Earlier quoted context omitted.

They do this via an iPhone like UA, though googlebot is still in there too.

I'm pretty sure that's just to see if the site is serving different content to mobile vs desktop. I think they also sometimes hit pages with no mention of googlebot.

If you do an nslookup on the IP, it should come up with crawl-xx-xx-xx-xx.googlebot.com, where xx-xx-xx-xx is the IP.

Re: How Google’s Web Crawler Bypasses Paywalls

#192
post #186

"Remember: Any time you introduce an access point for a trusted third party, you inevitably end up allowing access to anybody." See also: http://www.apple.com/customer-letter/ :)

Except of course what FBI is proposing wouldn't give access to "anybody". Just unfettered access for FBI, CIA and NSA by way of gag orders and national security letters, forcing Apple to break security of their hardware and not speak about it publicly. Of note is the fact that it wouldn't even give direct access to FBI, since they don't have the firmware keys (yet).

Yeah but if you don't trust those companies to be secure and to maintain that security throughout all their employees and contractors (see: Snowden). Then you must assume giving someone access is the same as giving everyone access.

Re: How Google’s Web Crawler Bypasses Paywalls

#193

Earlier quoted context omitted.

It isn't. But I have seen top internet companies (Netflix and the like) use it to authorise requests. It boggles my mind.

Curious where Netflix is using this to authorize requests. Any more info on that?

Their Android app wouldn't let you through if the device is unrecognised, that is, if your user agent has strings which Netflix hasn't whitelisted. They also check model ID, and build fingerprint too, I believe. A standard practice in the Android world, for some unknown reason. I don't know if they have stopped doing that.

Re: How Google’s Web Crawler Bypasses Paywalls

#194
post #186

"Remember: Any time you introduce an access point for a trusted third party, you inevitably end up allowing access to anybody." See also: http://www.apple.com/customer-letter/ :)

Except of course what FBI is proposing wouldn't give access to "anybody". Just unfettered access for FBI, CIA and NSA by way of gag orders and national security letters, forcing Apple to break security of their hardware and not speak about it publicly. Of note is the fact that it wouldn't even give direct access to FBI, since they don't have the firmware keys (yet).

In this example, the FBI is the "trusted third party", but by giving them access, we inevitably open access for everyone, as the system is no longer strongly secure. The trusted third party in the quote isn't asking for access for everybody either, but in the end that's what happens.

Re: How Google’s Web Crawler Bypasses Paywalls

#195
post #65
post #37

Earlier quoted context omitted.

Legally it might, I don't know. Morally, I'm not sure. 1. If they allowed all bots but disallowed all non-bots, that would raise questions of what defines a "bot". Couldn't I write a personal bot that fetches the story for me? As a browser addon, even? 2. It's even more complex since allowing bots means they allow tools that provide the information to third parties, as the bots are not intended for private use by the…

> If they allowed all bots but disallowed all non-bots, that would raise questions of what defines a "bot". Maybe, but I think its a pretty easy distinction. They aren't even allowing all bots - they're allowing a white list of them. You're not just writing your own bot to get around it, you're pretending to be someone else's bot. > But in practice, it seems they favor certain bots. Is it ok that the WSJ lets Google…

> If you view their website as a physical store / private establishment, then I assume that they have every right to establish who has access to what and under what conditions.

Not true. There are very specific laws about not being able to discriminate against protected classes of people.

Bars have to serve minorities, bakeries have to cater to same sex marriages, etc.

Where you draw the line of legislated equality within private property rights is pretty intriguing. I have don't have any answers, but lean heavily towards the libertarian bent.

Re: How Google’s Web Crawler Bypasses Paywalls

#196
post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

I know you were kidding, but I still don't understand this "exceeds access" bullshit. Here's how HTTP(S) works: I issue a REQUEST to the web server; the web RESPONDS with it, or denies it. It is up to the web server to respond or deny or do whatever it wants. If the web server is badly implemented or doesn't know what it's doing, it is the webserver's fault . Remember: it's just a request. I can request 100 dollars f…

And one more car analogy. If someone comes up to your car, pulls the door (issues a request) and based on the fact that it opens (responds 200), drives away in it, that would be seen as grand theft auto.

Whether the lock was implemented poorly or you just didn't lock it — doesn't matter.

Re: How Google’s Web Crawler Bypasses Paywalls

#197
post #83

Earlier quoted context omitted.

Unfortunately this will not help your defense. Andrew "Weev" Auernheimer was convicted of violating CFAA for exactly this (although the conviction was later overturned on a technicality). Again, exceeding authorized access means using your authorized access to obtain information you were not "entitled" to. So the question is not 'were you authorized' but rather it is 'were you entitled ' to that information? WTF 'ent…

> Andrew "Weev" Auernheimer was convicted of violating CFAA for exactly this (although the conviction was later overturned on a technicality). So, your example is...not an example? >WTF 'entitled' means is another question entirely No, in this case it is very clear: a request containing a particular user agent string is entitled. I have not tried this myself, but presumably you could verify that is the case by sendin…

> So, your example is...not an example?

At least in the US, the law doesn't work that way. Decisions will quite often cite some other similar case which reached the opposite conclusion, but under different circumstances, because that other case's decision says something like "X, if it weren't for Y" or "Fortunately for the defendant, they didn't Z, so not X", or something. That isn't binding precedent for the judge to apply X, but it's a very strong sign that X would be reasonable.

A court case that says "Yes, this violates CFAA but we have to throw out the case because A, B, and C" is very strong reason to believe that, if the next prosecutors avoid A, B, and C, the next judge will say "Yes, this still violates CFAA."

(IANAL but I read court cases because I find it useful to understand my jurisdiction's legal system.)

> a request containing a particular user agent string is entitled.

The phrasing of the law is very clear that the word "entitled" applies to a person, not to a request. Stealing someone's password and using their account is definitely a violation of CFAA (see e.g. http://www.wiggin.com/16332). In such a case, the account used to log in is quite plainly "entitled" / "authorized;" that's how you get the data. But the person logging in is not "entitled".

Re: How Google’s Web Crawler Bypasses Paywalls

#198
post #131

Earlier quoted context omitted.

Is that significantly different from viewing google's cached result of their legitimate access? The result is the same.

The difference is intent! The law says "authorized access" and the WSJ authorized Google to access their content in order to index it. The WSJ did not authorize that content to be presented, for free, to you, an end user, necessarily. I don't know which way a court would rule on it, but it's definitely not black-and-white. Sure, it's technically similar, but the court doesn't care. The court doesn't care if the law m…

I agree with you that this is what the law seems to think (being overly broad), but not if you are arguing this is sensical.

In other words, yes, the WSJ does intend to only give Google access to their content, and not the general public.

But no, the WSJ has not "authorized" Google by anything more official than a bank telling their security guards to let anyone into the vault who is wearing a blue t-shirt.

So yeah, I agree with you, there is a lot of conflation of technical means and the law, but we also shouldn't be granting to the WSJ that they are doing any real "authorizing" here, beyond wishing it and hoping it stays true.

Re: How Google’s Web Crawler Bypasses Paywalls

#200
1. Google's Web Crawlers are not "bypassing" paywall. It's the paywall that let's crawlers through. I.e. exactly the reverse of what the author implies with their headline.

2. The idea that this is somehow new is wrong. The way for a server to identify crawlers have "always" been to look at the user-agent, and, when done right, IP, verified either by net block owner or by doing PTR lookup and then checking that the A or AAA record for the claimed host points back at the same IPv4 or IPv6 address. Meanwhile, I do agree that paywalling is a more recent phenomenon, at least with regards to the extend it is popular among sites today, but the concept of presenting different data to crawlers and visitors arose much earlier and is something Google have been aware of and has made sure to delist such sites when found, whereas in fact Google has since then moved abit in the direction of allowing it in that they do so for Google News if declared as explained by others ITT.

So in my view, it seems that the author is jumping to incorrect conclusions based on an incomplete understanding of what's actually going on here. What then about the HN readership, how come this article became so highly voted and I don't see these issues raised by anyone else? Or maybe I'm just crazy?

Post reply on HN