Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google’s Web Crawler Bypasses Paywalls

elaineou.com

Post reply on HN
Print view View original

51–60 of 243 posts

Re: How Google’s Web Crawler Bypasses Paywalls

#51

Post by zaroth »

jrockway wrote: post #45
zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

User agent strings have a long history of being intentionally misleading. IE 11 claims to be "Mozilla/5.0". Chrome claims to be "Safari/537.36". The User-Agent string is all lies, and has been ever since the first site started doing UA sniffing.

It's intent that matters. Setting user-agent in order to properly render a page is legal. Setting a user-agent string to gain access to otherwise unauthorized content is probably not.

Re: How Google’s Web Crawler Bypasses Paywalls

#52

Post by ikeboy »

metafunctor wrote: post #41

I'm pretty sure Google will soon stop indexing WSJ. Why index something if the vast majority of users cannot access the pages behind the links? EDIT: The "paste a headline into Google" trick still works for me, though. If this continues to be the case, they will keep indexing, of course.

It doesn't work for me. They did say they're "testing" it, so maybe A/B testing conversion rates.

And yes, this violates Google's policies laid out explicitly at https://support.google.com/news/publisher/answer/40543?hl=en

Re: How Google’s Web Crawler Bypasses Paywalls

#56

Post by chinathrow »

eli wrote: post #10
crazysim wrote: post #4

Doesn't this kind of also hurt SEO? I'm would guess Google has some automated system to detect and apply a negative signal to sites that provide different content to a Googlebot user agent than a non-Googlebot user agent. I guess these sites are counting that the other signals outweigh that negative hit. Otherwise, why would expertsexchange be obligated to provide the answers at the very bottom? Did something change?

I'm 99% sure I've encountered a Googlebot crawling pages with the UA of a regular browser, presumably for exactly this purpose.

They do this via an iPhone like UA, though googlebot is still in there too.

Re: How Google’s Web Crawler Bypasses Paywalls

#57

Post by drostie »

nkrisc wrote: post #44
mchahn wrote: post #26

Bypassing the paywall is more unethical that blocking ads. It is one thing to have control over your own browser but another to steal something from another site. Also, isn't it illegal to bypass computer security?

How is modifying your own request headers any different than choosing to not display content returned in the response body? Their server can choose to do what it wants with your request and you can choose what to do with the response it sends. Are User-Agent headers legally protected identities?

Unfortunately, yes. The law legally protects anything which you might use to gain unauthorized access; that includes e.g. a password field. (That is, it is indeed breaking into a system if you type in the correct password, say by reading it on a post-it on someone's monitor, but you are not supposed to know that password.) This sort of thing makes the entire business of law complicated and impossible to automate.

Then again, lots of sci-fi dystopias are dreams of an automated law that somehow destroys the fabric of society, so...

Re: How Google’s Web Crawler Bypasses Paywalls

#58

Post by Joky »

metafunctor wrote: post #41

I'm pretty sure Google will soon stop indexing WSJ. Why index something if the vast majority of users cannot access the pages behind the links? EDIT: The "paste a headline into Google" trick still works for me, though. If this continues to be the case, they will keep indexing, of course.

Indexing is fine, a great feature would be if Google was able to show it only to the user that can access it.

Re: How Google’s Web Crawler Bypasses Paywalls

#60

Post by profeta »

zaroth wrote: post #51
jrockway wrote: post #45

Earlier quoted context omitted.

User agent strings have a long history of being intentionally misleading. IE 11 claims to be "Mozilla/5.0". Chrome claims to be "Safari/537.36". The User-Agent string is all lies, and has been ever since the first site started doing UA sniffing.

It's intent that matters. Setting user-agent in order to properly render a page is legal. Setting a user-agent string to gain access to otherwise unauthorized content is probably not.

intent can be many things.

i disable javascript. so i can't even see their page (and hence i don't know my access is being denied since i got a 200 http response, which means "OK") so i try different user agents with the intent of reading the content they are providing. Just like microsoft case.

Post reply on HN