Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google’s Web Crawler Bypasses Paywalls

elaineou.com

Post reply on HN
Print view View original

151–160 of 243 posts

Re: How Google’s Web Crawler Bypasses Paywalls

#151

Post by newjersey »

djsumdog wrote: post #142
comex wrote: post #116

Earlier quoted context omitted.

In any judge's opinion, you'd be full of shit. Judges aren't stupid. Which doesn't matter, because the WSJ is never going to sue you. But make sure you only consider your justification a personal one, not one that would provide any legal protection.

Most judges and legislators are fucking morons. Legal systems aren't a science. I wrote a thing on it a few years ago: http://khanism.org/security/legality/

I thought the problem was that it doesn't matter if WSJ is malicious in this case. Because cfaa is a criminal matter, zealous police departments and district attorney's offices can pursue heavy handed cases even without much cooperation from "the victim" which in this case is a corporation.

All I want to say is scrape cfaa now!

Re: How Google’s Web Crawler Bypasses Paywalls

#152

Post by oneeyedpigeon »

swanson wrote: post #40
zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

I wonder if a user-agent that was something like "Not a Googlebot" would a) allow access (probably regex based) and b) be truthful/plausible deniability.

What is your purpose in setting your user-agent string to that value, other than cleverly bypassing the paywall?

Re: How Google’s Web Crawler Bypasses Paywalls

#153

Post by lvs »

tomcam wrote: post #138
sneak wrote: post #123

Earlier quoted context omitted.

[I am not a lawyer and this is not legal advice.] It is not specifically defined in the law, so it reverts to the traditional meaning: anything the owner of the system says you aren't authorized to access. It's lunacy, I know. That's what HTTP headers and WAFs and such are for. But that's the stupid law, and it sent someone who used to be my friend to federal prison for changing a user agent and referrer and accessin…

Whoa. Care to tell that story or link to it?

I believe this refers to the Weev case.

Re: How Google’s Web Crawler Bypasses Paywalls

#154

Post by tajen »

njharman wrote: post #88
jsprogrammer wrote: post #63

Earlier quoted context omitted.

User-agent string is not an authorization mechanism.

Good luck arguing that in court. Esp against the multiple lawyers a corporation will be able to afford. If you don't get it, court doesn't care about what's reasonableness, technically correctness, etc. Only if your lawyers can convince jury/judge. Twinky made me crazy, It the gloves don't fit... and so forth.

If you don't set it for a particular website but generally browse the web with it, based on another legitimate purpose (I'm a developer, I had to test a website, I forgot the setting) could you oppose the court on "knowingly and with intent of defraud"? If you didn't see the paywall, how can it be "knowingly"?

Re: How Google’s Web Crawler Bypasses Paywalls

#155

Post by DanielBMarkham »

zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

As this entire thread has pointed out, the law makes no sense at all here. So you might as well say "Congratulations! You're going to jail! And I don't have to tell you why!" because none of the details matter anyway.

This is the spot we've reached through legislative meddling, and your best bet is either to be a good little consumer and don't make waves or do what you want and don't get caught. Neither of those seem to make a lot of sense in the long run.

So yes, congrats. It's jail for you -- but not for Google. Because they're Google, and you, well, you're not. You'd just better be happy we don't find out about you rooting your cellphone last year. Good grief.

ADD: I know you meant well, and I appreciated the tag, but there was something I didn't like about your comment. Now I know what it is. By making this a big deal, you're increasing the likelihood that this poor schmuck becomes the next "example" some federal prosecutor decides to make. It's not your fault, but it still sucks. Let's hope that doesn't happen.

Re: How Google’s Web Crawler Bypasses Paywalls

#156

Post by seanp2k2 »

ignoramous wrote: post #72
jsprogrammer wrote: post #63

Earlier quoted context omitted.

User-agent string is not an authorization mechanism.

It isn't. But I have seen top internet companies (Netflix and the like) use it to authorise requests. It boggles my mind.

Curious where Netflix is using this to authorize requests. Any more info on that?

Re: How Google’s Web Crawler Bypasses Paywalls

#157

Post by 1024core »

zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

I know you were kidding, but I still don't understand this "exceeds access" bullshit.

Here's how HTTP(S) works: I issue a REQUEST to the web server; the web RESPONDS with it, or denies it. It is up to the web server to respond or deny or do whatever it wants. If the web server is badly implemented or doesn't know what it's doing, it is the webserver's fault.

Remember: it's just a request. I can request 100 dollars from you; the fact that you give them to me does not make me a mugger.

Re: How Google’s Web Crawler Bypasses Paywalls

#158

Post by stonemetal »

zaroth wrote: post #33
ec109685 wrote: post #29

Earlier quoted context omitted.

Who has? Surely not the person who wrote the tutorial.

Even worse, the poor author has created a hacking tool capable of enabling said felony, which I believe could get them 10 or 20 years... I'm looking for the statute now. Edit: I was mis-remembering, the current law is against possession or manufacture of eavesdropping or wiretapping devices , not hacking tools. The EU has been playing with laws against hacking tools, but apparently nothing in the US yet against it. T…

Wouldn't it run afoul of the DMCA for making available a tool for circumvention?

Re: How Google’s Web Crawler Bypasses Paywalls

#159

Post by oneeyedpigeon »

mangeletti wrote: post #124

This is not meant to be purely controversial, but I thought long and hard about WSJ back a few months ago when HN mod (always forget his name) said to stop complaining about HN links being posted because paywalls were ok. I agree paywalls are ok. But some things are not ok. Take a look, for instance, at the WSJ.com home page with an ad blocker turned on (note all the missing letters and scrambled up titles). They wan…

> Take a look, for instance, at the WSJ.com home page with an ad blocker turned on (note all the missing letters and scrambled up titles).

I don't actually see what you're referring to; maybe its because I get redirected to http://www.wsj.com/europe. Maybe I have a different ad-blocker. Either way, it reminds me somewhat of NME's [1] homepage (New Musical Express, a popular music publication; not sure if it's really known outside of the UK). They deliver their images in such a way that they fall foul of my ad-blocker, although I haven't looked in enough depth to be certain whether this is a way of preventing ad-blockers, or purely unintentional.

[1] http://www.nme.com/

Re: How Google’s Web Crawler Bypasses Paywalls

#160

Post by maxerickson »

1024core wrote: post #157
zaroth wrote: post #27

And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…

I know you were kidding, but I still don't understand this "exceeds access" bullshit. Here's how HTTP(S) works: I issue a REQUEST to the web server; the web RESPONDS with it, or denies it. It is up to the web server to respond or deny or do whatever it wants. If the web server is badly implemented or doesn't know what it's doing, it is the webserver's fault . Remember: it's just a request. I can request 100 dollars f…

If you request the 100 dollars with a specially crafted piece of paper that I glance at and believe is legitimate, you are committing a crime (check fraud). Which whatever, let's not focus on the analogy. Whether you like it or not, the law doesn't necessarily view the valid server response as authorization to access the requested url, it examines what you were thinking when you created the url.
Post reply on HN