Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Google’s Web Crawler Bypasses Paywalls

elaineou.com

Post reply on HN
Print view View original

221–230 of 243 posts

Re: How Google’s Web Crawler Bypasses Paywalls

#221

Post by germanier »

acomjean wrote: post #39
matt_wulfeck wrote: post #15

I like wsj but I only read maybe 1 article every other day. They need a more reasonable price point, especially since the market will almost bear no price at all. That being said I do enjoy their content, save for maybe the op-eds.

I'm surprised that most online papers won't sell you one day's worth online for a buck or so. Like buying a real newspaper. They all seem to want to sell subscriptions, which are perpetual and probably difficult to cancel..

You can buy single WSJ print or online articles for 29 cents on Blendle or today's whole paper for 3.20.

I don't know if it's available in the US yet but they are at least planning to launch in the near future.

Re: How Google’s Web Crawler Bypasses Paywalls

#222

Post by pbhjpbhj »

TheDong wrote: post #140
fiatmoney wrote: post #133

Earlier quoted context omitted.

If I run a car wash with a sign, "red cars washed free", and you paint your car red, have you defrauded me? You're not really a red car, you're just pretending !

Terrible analogy. You're talking about a sign with practically no legal meaning. This discussion is about a law which does have legal meaning. To fix your analogy, it would be "If there's a law that says only explicitly authorized cars are allowed in a car wash; else 5 years in prison, and I have a car wash and say 'only red cars' ...". Of course, that still doesn't properly capture this since the intent of the law a…

The law seems to use trespass as the analogue for unauthorised access. We can argue it's a bad analogy, but it's the one we have.

The door is locked, the key is right there and all you have to do is pick it up and you can gain access, but that doesn't make you authorised to gain access.

The CFAA and UK CMA may well be overzealous in restricting what should (to technologists) be allowed, but it's what we have as statute in the respective jurisdictions.

Re: How Google’s Web Crawler Bypasses Paywalls

#223

Post by tomkwok »

obelisk_ wrote: post #200

1. Google's Web Crawlers are not "bypassing" paywall. It's the paywall that let's crawlers through. I.e. exactly the reverse of what the author implies with their headline. 2. The idea that this is somehow new is wrong. The way for a server to identify crawlers have "always" been to look at the user-agent, and, when done right, IP, verified either by net block owner or by doing PTR lookup and then checking that the A…

> Google's Web Crawlers are not "bypassing" paywall. It's the paywall that let's crawlers through. I.e. exactly the reverse of what the author implies with their headline.

Don't nitpick. It's just a shortened version of How To "Be" a Google’s Web Crawler to Bypass Paywalls. You get it. I get it. Everyone gets it.

Re: How Google’s Web Crawler Bypasses Paywalls

#225

Post by njharman »

zodiac wrote: post #126
njharman wrote: post #92

Earlier quoted context omitted.

No content is being given or taken. This is restriction on distribution / copying. Massively different ethics than stealing despite what the Copyright Cabal wants you to think. The only loss is the energy/bandwidth/cycles WSJ servers spent answering your request. Which, I believe, has been basis of computer "fraud" cases.

> Which, I believe, has been basis of computer "fraud" cases. This can't be true. Surely the argument for why, say, a WSJ-paywall-bypassing-tool causes damage (in the legal sense) to WSJ is that it allows people who would otherwise pay for content to get it for free, thus depriving WSJ of income. Moreover, I don't think prosecutors need to prove that you caused harm in order to charge you with computer fraud, since,…

Yeah I spent 10min looking and couldn't find anything. Maybe I'm remembering a "what if" from early days cypherpunk stuff.

Prosecutors never (want to) just charge one thing. They want a laundry list of dozen or more crimes so they can coerce suspect into pleading guilty. The, "theft of resources" would just add to the pile.

Re: How Google’s Web Crawler Bypasses Paywalls

#226

Post by njharman »

tajen wrote: post #154
njharman wrote: post #88

Earlier quoted context omitted.

Good luck arguing that in court. Esp against the multiple lawyers a corporation will be able to afford. If you don't get it, court doesn't care about what's reasonableness, technically correctness, etc. Only if your lawyers can convince jury/judge. Twinky made me crazy, It the gloves don't fit... and so forth.

If you don't set it for a particular website but generally browse the web with it, based on another legitimate purpose (I'm a developer, I had to test a website, I forgot the setting) could you oppose the court on "knowingly and with intent of defraud"? If you didn't see the paywall, how can it be "knowingly"?

Trot out expert witnesses (on web dev). "Sir, have you ever "forgot" the setting?" No, and it's ridculous to think anyone would. "Is it generally know by web devs the dangers and circumvention ability of this setting" Absolutely. etc.

"Here is a transcript of electronic forum detailing how to circumvent access controls and defraud the victim using the exact methods defendant used to access victim's website. A forum the defendent heavily traffics. Often multiple times per day."

Ladies and gentlmen I ask you is it more likely that the defendent, a self professed developer, and expert in these circumvention methods, who reguralry participates on forums discussing hacking and defrauding companies such as the victim. I ask you is it reasonable to believe he "just forgot"?

Lawyers man, Lawyers! Can you not understand that rationality, technicallity don't matter. Lawyering is like statistics/graphs. You can get the data to say whatever you want.

Re: How Google’s Web Crawler Bypasses Paywalls

#227

Post by jsprogrammer »

jsprogrammer wrote: post #205
zaroth wrote: post #174

Earlier quoted context omitted.

It's the best example we got. The case was overturned (after he spent quite some time in federal prison) not because it was found that he didn't violate the CFAA but because the charges were brought in the wrong jurisdiction. Again I think you're confusing the fact someone could trick the server into delivering the content for free with WSJ intending to deliver their content to you for free. Since WSJ clearly intends…

As far as the law is concerned, he did not violate anything. You do not have to prove yourself innocent, the burden is on the prosecutor to prove a violation. In the example you cite, no violation has been shown. It is not at all clear that WSJ intends Googlebot to get their content for free while others must pay. Thus is actually against Google's policies, which would actually call into question whether WSJ's behavi…

If you will down mod, at least show how I am wrong!

Re: How Google’s Web Crawler Bypasses Paywalls

#229

Post by jsprogrammer »

njharman wrote: post #226
tajen wrote: post #154

Earlier quoted context omitted.

If you don't set it for a particular website but generally browse the web with it, based on another legitimate purpose (I'm a developer, I had to test a website, I forgot the setting) could you oppose the court on "knowingly and with intent of defraud"? If you didn't see the paywall, how can it be "knowingly"?

Trot out expert witnesses (on web dev). "Sir, have you ever "forgot" the setting?" No, and it's ridculous to think anyone would. "Is it generally know by web devs the dangers and circumvention ability of this setting" Absolutely. etc. "Here is a transcript of electronic forum detailing how to circumvent access controls and defraud the victim using the exact methods defendant used to access victim's website. A forum t…

User agent string is not an access control mechanism.

Re: How Google’s Web Crawler Bypasses Paywalls

#230

Post by majewsky »

vonklaus wrote: post #211
yyin wrote: post #209

Earlier quoted context omitted.

There are certainly more you omitted (similar to your code CDN for javascipt, fonts, etc.)... consider something like recaptcha. In my experience, all those requests to api.recaptcha.net get forwarded to "www.google.com" My experience has been that if a user for whatever reason cannot access the IP du jour for www.google.com (www.google.[cctld] will not suffice) then that user is prevented from using the myriad websi…

Right, that was my point. I actually did include code CDN and fonts in the original list, but regardless I think google is an awesome company but as a community it is just downright irresponsible to fork this level of control to an entity. While I consider Snowden a proper hero it is almost a certainty that this could happen to a "friendly" entity like google. In that, the NSA likely has some top programmers who coul…

> as search deteriorates

Can you expand on why search is deteriorating? Honest question. I certainly don't see the relevance of search sinking, nor can I see any competitor in the market that could even come close to threatening Google's monopoly on search.

Post reply on HN