If you hit a paywall or a "sign up to access this content" message from a google search result, report it. Google will remove them from the search results, they will lose their largest traffic source, and they will address the issue. Or they won't because they have enough paying customers.
How Google’s Web Crawler Bypasses Paywalls
121–130 of 243 posts
Re: How Google’s Web Crawler Bypasses Paywalls
#122Earlier quoted context omitted.
You are confusing an engineering process with a legal one. Unfortunately, when the prosecutors come, they will only care about the legal process.
It's not engineering, it's common sense.
Common sense is not a set of legal procedures and rules either.
The legal world cares about how the law applies to the facts of the case, not about how common sense applies.
Not saying I like it.
Re: How Google’s Web Crawler Bypasses Paywalls
#123Earlier quoted context omitted.
It's intent that matters. Setting user-agent in order to properly render a page is legal. Setting a user-agent string to gain access to otherwise unauthorized content is probably not.
Unauthorized? If you put up a sign and tell people they have to pay to look at it, is it illegal to look at it and not pay? This should be a rhetorical question.
[I am not a lawyer and this is not legal advice.]
It is not specifically defined in the law, so it reverts to the traditional meaning: anything the owner of the system says you aren't authorized to access.
It's lunacy, I know. That's what HTTP headers and WAFs and such are for. But that's the stupid law, and it sent someone who used to be my friend to federal prison for changing a user agent and referrer and accessing unprotected data on the web.
Tread carefully.
Re: How Google’s Web Crawler Bypasses Paywalls
#124This is not meant to be purely controversial, but I thought long and hard about WSJ back a few months ago when HN mod (always forget his name) said to stop complaining about HN links being posted because paywalls were ok. I agree paywalls are ok. But some things are not ok.
Take a look, for instance, at the WSJ.com home page with an ad blocker turned on (note all the missing letters and scrambled up titles). They want me to pay, and they want me to see ads, and they want to track my behavior? Should I send them my DNA also?
Organizations like WSJ are exactly the disease that causes ad blockers to proliferate and ruin the web for all the decent publishers. They're at war with my privacy (by breaking their site intentionally when I visit with a blocker on). They want it all, ads, tracking, your private data, and subscription revenue, not to mention...
# Agenda-Driven Content
I mean, we're basically talking about NBC or Fox here, just on the web. Imagine every morning when you woke up you turned on the television and tune to some "news" show. After talking about the weather, they start talking about a lost pickle that is thought to be potentially alive and moving about with free will. Over the next two years, talk about the same pickle extends to every other TV show. Before you know it, everybody in the nation is talking about the same pickle. Years go by, and that pickle has become a part of our society, and that's not because people are born with an innate care the well-being of pickles, but because "news" shows taught them to be.
That's not a good position to be in. I have to believe I'm not the only one in here that doesn't watch any TV. So, why do we all treat the same media giants differently on the web? We crave their content so much that we build browser add-ons to get to their content, etc.
Re: How Google’s Web Crawler Bypasses Paywalls
#125Bypassing the paywall is more unethical that blocking ads. It is one thing to have control over your own browser but another to steal something from another site. Also, isn't it illegal to bypass computer security?
How is modifying your own request headers any different than choosing to not display content returned in the response body? Their server can choose to do what it wants with your request and you can choose what to do with the response it sends. Are User-Agent headers legally protected identities?
The difference is substantial; in principle, one can modify his request to intentionally bypass the authorization mechanism occurring on the server. One cannot mislead anyone/anything by displaying his data in a customized way in private on his computer.
Re: How Google’s Web Crawler Bypasses Paywalls
#126Am I alone in feeling like this is akin to a tutorial on how you can shoplift without getting caught? WSJ, for better or worse, does not want to give you content without your paying for it. If you take that content without paying, you are stealing. Just because you have figured out how to get past their security does not mean it's not stealing. (See the second precept here: https://en.wikipedia.org/wiki/Five_Precepts…
No content is being given or taken. This is restriction on distribution / copying. Massively different ethics than stealing despite what the Copyright Cabal wants you to think. The only loss is the energy/bandwidth/cycles WSJ servers spent answering your request. Which, I believe, has been basis of computer "fraud" cases.
> Which, I believe, has been basis of computer "fraud" cases.
This can't be true. Surely the argument for why, say, a WSJ-paywall-bypassing-tool causes damage (in the legal sense) to WSJ is that it allows people who would otherwise pay for content to get it for free, thus depriving WSJ of income.
Moreover, I don't think prosecutors need to prove that you caused harm in order to charge you with computer fraud, since, for example, CFAA falls under criminal law.
Re: How Google’s Web Crawler Bypasses Paywalls
#127This is an odd debate. Let's say a restaurant declares "veterans eat free." This blog post is like a friend telling you "Hey if you tell this restaurant you're a vet they'll give you a free meal." No one said it's legal or ethical. It's lying to trick someone into giving you something at their expense.
I think the relevant point, underscored by the author's last sentence, is it doesn't matter who you open a back door for - it opens the possibility for anyone to barge through.
Re: How Google’s Web Crawler Bypasses Paywalls
#128And congratulations, you have likely just "exceeded authorized access" and committed a felony violation of the CFAA punishable by a fine or imprisonment for not more than 5 years under 18 U.S.C. § 1030(c)(2)(B)(i). From the ABA: "Exceeds authorized access is defined in the Computer Fraud and Abuse Act (CFAA) to mean "to access a computer with authorization and to use such access to obtain or alter information in the…
Doesn't this just prove that CFAA (and similar laws in other jurisdictions) is massively over-broad and needs to be narrowed. I'm surprised there is not more campaigning for CFAA repeal and replacement with far narrower legislation.
If someone hasn't exploited a security bug (I mean a real security bug, like a buffer overflow, I don't consider behaviour such as serving up content to certain User-Agents only a genuine "security bug"), and they haven't bruteforced/cracked/acquired a password or private key, and they aren't sending unreasonable amounts of traffic ((D)DOS), it should not be a crime, and the law should be changed to reflect that principle. The law should reflect the common sense of the technically literate, but it doesn't, because it was written by the technically illiterate.
Re: How Google’s Web Crawler Bypasses Paywalls
#129Earlier quoted context omitted.
How is modifying your own request headers any different than choosing to not display content returned in the response body? Their server can choose to do what it wants with your request and you can choose what to do with the response it sends. Are User-Agent headers legally protected identities?
The difference is substantial; in principle, one can modify his request to intentionally bypass the authorization mechanism occurring on the server. One cannot mislead anyone/anything by displaying his data in a customized way in private on his computer.
Fair point. Intent matters. However must it be demonstrated that there is some specific malicious intent behind changing a request header versus simply changing your user agent for the heck of it?
In a related point, some news sites load a modal and prevent scrolling over an article asking you to sign up. However if the full article is included in the response and I read it by simply viewing the response body (HTML) is that circumventing security? Actual example. In this case modifying how the response is rendered in my browser, I can bypass their intentions.
Obviously the better way of doing this would be to not send the entire article content until they've determined I should be able to view it.
Re: How Google’s Web Crawler Bypasses Paywalls
#130This is not meant to be purely controversial, but I thought long and hard about WSJ back a few months ago when HN mod (always forget his name) said to stop complaining about HN links being posted because paywalls were ok. I agree paywalls are ok. But some things are not ok. Take a look, for instance, at the WSJ.com home page with an ad blocker turned on (note all the missing letters and scrambled up titles). They wan…
If they make sending your DNA a requirement of consuming their content, then yes, you send it to them if you want their content. That's their right, as owners of something, to dictate its use.
You aren't entitled to WSJ.com, NBC, or Fox.