Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google’s Web Crawler Bypasses Paywalls

elaineou.com

Post reply on HN
Print view View original

71–80 of 243 posts

Re: How Google’s Web Crawler Bypasses Paywalls

#71

Post by ikeboy »

mod wrote: post #61
ikeboy wrote: post #50

Earlier quoted context omitted.

WSJ is violating Google's rules referenced here: https://support.google.com/news/publisher/answer/40543?hl=en

So? Is this a "two wrongs make a right?" More importantly, is violating "Google's rules" suddenly a violation of law?

This would never go to court, but if it did, I could see a jury being sympathetic. You could sort of argue that WSJ had an obligation to Google to allow referred users to view the pages, and that Google passed that right onto the user, so you were contractually authorized to access it.

IANAL and this is most likely wrong, but kind of plausible to my NAL mind.

Re: How Google’s Web Crawler Bypasses Paywalls

#72

Post by ignoramous »

jsprogrammer wrote: post #63
zaroth wrote: post #51

Earlier quoted context omitted.

It's intent that matters. Setting user-agent in order to properly render a page is legal. Setting a user-agent string to gain access to otherwise unauthorized content is probably not.

User-agent string is not an authorization mechanism.

It isn't. But I have seen top internet companies (Netflix and the like) use it to authorise requests. It boggles my mind.

Re: How Google’s Web Crawler Bypasses Paywalls

#76

Post by zaroth »

jsprogrammer wrote: post #63
zaroth wrote: post #51

Earlier quoted context omitted.

It's intent that matters. Setting user-agent in order to properly render a page is legal. Setting a user-agent string to gain access to otherwise unauthorized content is probably not.

User-agent string is not an authorization mechanism.

Interestingly, the CFAA does not define the term "without authorization" however it does define "exceeding authorization" exactly as I quoted above;

  - to access a computer with authorization and to use such access
    to obtain or alter information in the computer that the accesser
    is not entitled so to obtain or alter.
So arguing User-agent is not an authorization mechanism probably won't help you, because exceeding authorization means, first, that you were authorized to access the computer (HTTP GET returns 200) but then that you used that access to obtain information in the computer that you were "not entitled so to obtain."

Re: How Google’s Web Crawler Bypasses Paywalls

#77

Post by jsprogrammer »

zaroth wrote: post #76
jsprogrammer wrote: post #63

Earlier quoted context omitted.

User-agent string is not an authorization mechanism.

Interestingly, the CFAA does not define the term "without authorization" however it does define "exceeding authorization" exactly as I quoted above; - to access a computer with authorization and to use such access to obtain or alter information in the computer that the accesser is not entitled so to obtain or alter. So arguing User-agent is not an authorization mechanism probably won't help you, because exceeding aut…

If the computer on the public Internet responds to a standard HTTP request, it has explicitly authorized my access to whatever information it sent me.

Re: How Google’s Web Crawler Bypasses Paywalls

#78

Post by dheera »

zaroth wrote: post #51
jrockway wrote: post #45

Earlier quoted context omitted.

User agent strings have a long history of being intentionally misleading. IE 11 claims to be "Mozilla/5.0". Chrome claims to be "Safari/537.36". The User-Agent string is all lies, and has been ever since the first site started doing UA sniffing.

It's intent that matters. Setting user-agent in order to properly render a page is legal. Setting a user-agent string to gain access to otherwise unauthorized content is probably not.

In that case, I'm just going to set my User-Agent permanently to Google's crawler for fun and see how the web renders, in general. Just for fun, because I want to see what the world looks like from a different perspective. That's my intent. I've stated it. Now that my user agent has been changed, I'm going to go grab lunch, carry on with life, maybe check out some cat pictures and then maybe some news sites over tea and snacks, and then go for a run.

Re: How Google’s Web Crawler Bypasses Paywalls

#79

Post by mattmaroon »

coverband wrote: post #54

My Windows anti-virus deletes the linked sample code automatically upon download, marking it as "Trojan:Win32/Spursint.A". Did anyone have the same experience? (I was actually more interested in using it as a template for writing a simple Chrome extension.)

Yep. I then pasted it but it didn't work on wsj.com. Oh well.

Re: How Google’s Web Crawler Bypasses Paywalls

#80

Post by matt_wulfeck »

acomjean wrote: post #39
matt_wulfeck wrote: post #15

I like wsj but I only read maybe 1 article every other day. They need a more reasonable price point, especially since the market will almost bear no price at all. That being said I do enjoy their content, save for maybe the op-eds.

I'm surprised that most online papers won't sell you one day's worth online for a buck or so. Like buying a real newspaper. They all seem to want to sell subscriptions, which are perpetual and probably difficult to cancel..

Even a dollar a day. I can pay Netflix $10 a month and stream unlimited HD video, but wsj wants $30 to read the first few paragraphs of a few articles a day?

The pricing here is much too aggressive

Post reply on HN