Earlier quoted context omitted.
They do this via an iPhone like UA, though googlebot is still in there too.
I'm pretty sure that's just to see if the site is serving different content to mobile vs desktop. I think they also sometimes hit pages with no mention of googlebot.
How Google’s Web Crawler Bypasses Paywalls
191–200 of 243 posts
Re: How Google’s Web Crawler Bypasses Paywalls
#192"Remember: Any time you introduce an access point for a trusted third party, you inevitably end up allowing access to anybody." See also: http://www.apple.com/customer-letter/ :)
Except of course what FBI is proposing wouldn't give access to "anybody". Just unfettered access for FBI, CIA and NSA by way of gag orders and national security letters, forcing Apple to break security of their hardware and not speak about it publicly. Of note is the fact that it wouldn't even give direct access to FBI, since they don't have the firmware keys (yet).
Re: How Google’s Web Crawler Bypasses Paywalls
#193Earlier quoted context omitted.
It isn't. But I have seen top internet companies (Netflix and the like) use it to authorise requests. It boggles my mind.
Curious where Netflix is using this to authorize requests. Any more info on that?
Re: How Google’s Web Crawler Bypasses Paywalls
#194"Remember: Any time you introduce an access point for a trusted third party, you inevitably end up allowing access to anybody." See also: http://www.apple.com/customer-letter/ :)
Except of course what FBI is proposing wouldn't give access to "anybody". Just unfettered access for FBI, CIA and NSA by way of gag orders and national security letters, forcing Apple to break security of their hardware and not speak about it publicly. Of note is the fact that it wouldn't even give direct access to FBI, since they don't have the firmware keys (yet).
Re: How Google’s Web Crawler Bypasses Paywalls
#195Earlier quoted context omitted.
Legally it might, I don't know. Morally, I'm not sure. 1. If they allowed all bots but disallowed all non-bots, that would raise questions of what defines a "bot". Couldn't I write a personal bot that fetches the story for me? As a browser addon, even? 2. It's even more complex since allowing bots means they allow tools that provide the information to third parties, as the bots are not intended for private use by the…
> If they allowed all bots but disallowed all non-bots, that would raise questions of what defines a "bot". Maybe, but I think its a pretty easy distinction. They aren't even allowing all bots - they're allowing a white list of them. You're not just writing your own bot to get around it, you're pretending to be someone else's bot. > But in practice, it seems they favor certain bots. Is it ok that the WSJ lets Google…
Not true. There are very specific laws about not being able to discriminate against protected classes of people.
Bars have to serve minorities, bakeries have to cater to same sex marriages, etc.
Where you draw the line of legislated equality within private property rights is pretty intriguing. I have don't have any answers, but lean heavily towards the libertarian bent.
Re: How Google’s Web Crawler Bypasses Paywalls
#196And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…
I know you were kidding, but I still don't understand this "exceeds access" bullshit. Here's how HTTP(S) works: I issue a REQUEST to the web server; the web RESPONDS with it, or denies it. It is up to the web server to respond or deny or do whatever it wants. If the web server is badly implemented or doesn't know what it's doing, it is the webserver's fault . Remember: it's just a request. I can request 100 dollars f…
Whether the lock was implemented poorly or you just didn't lock it — doesn't matter.
Re: How Google’s Web Crawler Bypasses Paywalls
#197Earlier quoted context omitted.
Unfortunately this will not help your defense. Andrew "Weev" Auernheimer was convicted of violating CFAA for exactly this (although the conviction was later overturned on a technicality). Again, exceeding authorized access means using your authorized access to obtain information you were not "entitled" to. So the question is not 'were you authorized' but rather it is 'were you entitled ' to that information? WTF 'ent…
> Andrew "Weev" Auernheimer was convicted of violating CFAA for exactly this (although the conviction was later overturned on a technicality). So, your example is...not an example? >WTF 'entitled' means is another question entirely No, in this case it is very clear: a request containing a particular user agent string is entitled. I have not tried this myself, but presumably you could verify that is the case by sendin…
At least in the US, the law doesn't work that way. Decisions will quite often cite some other similar case which reached the opposite conclusion, but under different circumstances, because that other case's decision says something like "X, if it weren't for Y" or "Fortunately for the defendant, they didn't Z, so not X", or something. That isn't binding precedent for the judge to apply X, but it's a very strong sign that X would be reasonable.
A court case that says "Yes, this violates CFAA but we have to throw out the case because A, B, and C" is very strong reason to believe that, if the next prosecutors avoid A, B, and C, the next judge will say "Yes, this still violates CFAA."
(IANAL but I read court cases because I find it useful to understand my jurisdiction's legal system.)
> a request containing a particular user agent string is entitled.
The phrasing of the law is very clear that the word "entitled" applies to a person, not to a request. Stealing someone's password and using their account is definitely a violation of CFAA (see e.g. http://www.wiggin.com/16332). In such a case, the account used to log in is quite plainly "entitled" / "authorized;" that's how you get the data. But the person logging in is not "entitled".
Re: How Google’s Web Crawler Bypasses Paywalls
#198Earlier quoted context omitted.
Is that significantly different from viewing google's cached result of their legitimate access? The result is the same.
The difference is intent! The law says "authorized access" and the WSJ authorized Google to access their content in order to index it. The WSJ did not authorize that content to be presented, for free, to you, an end user, necessarily. I don't know which way a court would rule on it, but it's definitely not black-and-white. Sure, it's technically similar, but the court doesn't care. The court doesn't care if the law m…
In other words, yes, the WSJ does intend to only give Google access to their content, and not the general public.
But no, the WSJ has not "authorized" Google by anything more official than a bank telling their security guards to let anyone into the vault who is wearing a blue t-shirt.
So yeah, I agree with you, there is a lot of conflation of technical means and the law, but we also shouldn't be granting to the WSJ that they are doing any real "authorizing" here, beyond wishing it and hoping it stays true.
Re: How Google’s Web Crawler Bypasses Paywalls
#199Re: How Google’s Web Crawler Bypasses Paywalls
#2002. The idea that this is somehow new is wrong. The way for a server to identify crawlers have "always" been to look at the user-agent, and, when done right, IP, verified either by net block owner or by doing PTR lookup and then checking that the A or AAA record for the claimed host points back at the same IPv4 or IPv6 address. Meanwhile, I do agree that paywalling is a more recent phenomenon, at least with regards to the extend it is popular among sites today, but the concept of presenting different data to crawlers and visitors arose much earlier and is something Google have been aware of and has made sure to delist such sites when found, whereas in fact Google has since then moved abit in the direction of allowing it in that they do so for Google News if declared as explained by others ITT.
So in my view, it seems that the author is jumping to incorrect conclusions based on an incomplete understanding of what's actually going on here. What then about the HN readership, how come this article became so highly voted and I don't see these issues raised by anyone else? Or maybe I'm just crazy?