Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google’s Web Crawler Bypasses Paywalls

elaineou.com

Post reply on HN
Print view View original

161–170 of 243 posts

Re: How Google’s Web Crawler Bypasses Paywalls

#161

Post by barrkel »

1024core wrote: post #157
zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

I know you were kidding, but I still don't understand this "exceeds access" bullshit. Here's how HTTP(S) works: I issue a REQUEST to the web server; the web RESPONDS with it, or denies it. It is up to the web server to respond or deny or do whatever it wants. If the web server is badly implemented or doesn't know what it's doing, it is the webserver's fault . Remember: it's just a request. I can request 100 dollars f…

What if you tell me my car's broken when it's not and request 100 dollars from me to fix it? That would be fraud.

So you go up to a server and lie to it, and it gives you something; is that not acquiring things through deception?

The structure of your argument suggests that e.g. breaking into an ssh server by issuing a login request with a known password which it responds to, isn't illegal. And further, that if data is acquired from the server, there is still no crime - the ssh protocol too is just requests to the server, it's all bits down the line. It's clearly nonsense.

Re: How Google’s Web Crawler Bypasses Paywalls

#163

Post by 0xffff2 »

mangeletti wrote: post #124

This is not meant to be purely controversial, but I thought long and hard about WSJ back a few months ago when HN mod (always forget his name) said to stop complaining about HN links being posted because paywalls were ok. I agree paywalls are ok. But some things are not ok. Take a look, for instance, at the WSJ.com home page with an ad blocker turned on (note all the missing letters and scrambled up titles). They wan…

>Take a look, for instance, at the WSJ.com home page with an ad blocker turned on (note all the missing letters and scrambled up titles)

Uh, what? Using uBlock origin, when I visit wsj.com I get what looks like a perfectly normal page. Nothing is scrambled at all.

Re: How Google’s Web Crawler Bypasses Paywalls

#164

Post by cookiecaper »

lvs wrote: post #153
tomcam wrote: post #138

Earlier quoted context omitted.

Whoa. Care to tell that story or link to it?

I believe this refers to the Weev case.

It may, but weev is probably not the only person whose been put away for that. This type of activity is the basis for Google and many other tech startups. I hope they catch Larry and Sergei soon, they've been on the lam for almost two decades!

Re: How Google’s Web Crawler Bypasses Paywalls

#165

Post by jsprogrammer »

zaroth wrote: post #83
jsprogrammer wrote: post #77

Earlier quoted context omitted.

If the computer on the public Internet responds to a standard HTTP request, it has explicitly authorized my access to whatever information it sent me.

Unfortunately this will not help your defense. Andrew "Weev" Auernheimer was convicted of violating CFAA for exactly this (although the conviction was later overturned on a technicality). Again, exceeding authorized access means using your authorized access to obtain information you were not "entitled" to. So the question is not 'were you authorized' but rather it is 'were you entitled ' to that information? WTF 'ent…

> Andrew "Weev" Auernheimer was convicted of violating CFAA for exactly this (although the conviction was later overturned on a technicality).

So, your example is...not an example?

>WTF 'entitled' means is another question entirely

No, in this case it is very clear: a request containing a particular user agent string is entitled. I have not tried this myself, but presumably you could verify that is the case by sending a request with the appropriate user agent.

Re: How Google’s Web Crawler Bypasses Paywalls

#166

Post by lstamour »

slig wrote: post #21
lstamour wrote: post #16

Earlier quoted context omitted.

Actually I noticed sites have simply changed policies -- if you're a regular visitor your cookies will identify you and block content. The Incognito mode trick works for WSJ and others that would still check the referrer header. Allowing Googlebot access and checking the referrer header are two different things. Also, Google has published IP addresses it uses, so this extension might not last long...

> Also, Google has published IP addresses it uses, so this extension might not last long... They do not [1], but you can find out by doing a reverse DNS query. > "Google doesn't post a public list of IP addresses for webmasters to whitelist" [1] https://support.google.com/webmasters/answer/80553?hl=en

Sorry, that was what I meant. :)

Re: How Google’s Web Crawler Bypasses Paywalls

#167

Post by mark-r »

mikestew wrote: post #107

So does HN now choose to not post articles from the WSJ? I was comfortable with the "google it" trick, and frankly was a little annoyed with constant "paywall, wah!" comments when what should be by now a well-known workaround was available. But that workaround no longer works.

They've been testing the new wall for a while now. I know I made one of those "paywall wah" comments when the Google workaround didn't work for me. Then the next time I tried it worked fine, so it must have been random selection.

Re: How Google’s Web Crawler Bypasses Paywalls

#168

Post by jsprogrammer »

compiler-guy wrote: post #122
CyberDildonics wrote: post #112

Earlier quoted context omitted.

It's not engineering, it's common sense.

Common sense is not a set of legal procedures and rules either. The legal world cares about how the law applies to the facts of the case, not about how common sense applies. Not saying I like it.

The facts are dictated by the engineering. Is a lawyer a computer networks expert? Not by default. They will need to defer to the engineers.

Re: How Google’s Web Crawler Bypasses Paywalls

#169

Post by type0 »

zodiac wrote: post #126
njharman wrote: post #92

Earlier quoted context omitted.

No content is being given or taken. This is restriction on distribution / copying. Massively different ethics than stealing despite what the Copyright Cabal wants you to think. The only loss is the energy/bandwidth/cycles WSJ servers spent answering your request. Which, I believe, has been basis of computer "fraud" cases.

> Which, I believe, has been basis of computer "fraud" cases. This can't be true. Surely the argument for why, say, a WSJ-paywall-bypassing-tool causes damage (in the legal sense) to WSJ is that it allows people who would otherwise pay for content to get it for free, thus depriving WSJ of income. Moreover, I don't think prosecutors need to prove that you caused harm in order to charge you with computer fraud, since,…

> it allows people who would otherwise pay for content to get it for free

But the sole existence of this trick and the person that would use it is exactly someone who would not pay, therefore your argument does not hold. And stealing it is not, it is more like listening to the outdoor rock concert beside the fence because you don't want to pay, inconvenient- sure, so plenty of people would still pay.

Cloaking is against Googles rules so it is WSJ that's dishonest.

Re: How Google’s Web Crawler Bypasses Paywalls

#170

Post by bigjust »

newjersey wrote: post #151
djsumdog wrote: post #142

Earlier quoted context omitted.

Most judges and legislators are fucking morons. Legal systems aren't a science. I wrote a thing on it a few years ago: http://khanism.org/security/legality/

I thought the problem was that it doesn't matter if WSJ is malicious in this case. Because cfaa is a criminal matter, zealous police departments and district attorney's offices can pursue heavy handed cases even without much cooperation from "the victim" which in this case is a corporation. All I want to say is scrape cfaa now!

> scrape cfaa now!

pun?

Post reply on HN