Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

cdn.ca9.uscourts.gov

Post reply on HN
Print view View original

221–230 of 293 posts

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#221

Post by TeMPOraL »

dickjocke wrote: post #78
meritt wrote: post #64

Earlier quoted context omitted.

If you have to login to access the data, that data isn't publicly available, and unlikely to be protected by this decision. Page 31 of the filing specifically differentiates this case from one regarding Facebook: "While Power Ventures was gathering user data that was protected by Facebook’s username and password authentication system, the data hiQ was scraping was available to anyone with a web browser."

Wait, doesn't Linkedin have pretty strong authentication safeuguards to view user profiles. if you google someone and click their linkedin without being logged in to LinkedIn, you're always directed to sign up or log in to it. Twitter is the only example I can think of among the large social media sites that doesn't require you to be logged in to see profiles

It's not authentication safeguards, it's dark UX patterns driving you to signup. Whether or not you'll see someone's profile and how much of it you'll see depends on how they fingerprint you, where you navigate from, and on the phase of the Moon. LinkedIn is notorious for that.

My memory might be misleading me here, but I think I remember years ago in one case I could see less info about a profile when logged in to my account than I could see while logged out and navigating from a Google search...

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#222

Post by TeMPOraL »

leesalminen wrote: post #116
cwkoss wrote: post #104

Earlier quoted context omitted.

Creating barriers to entry is an antisocial tactic that harms consumers and society at large. It is the responsibility of moral consumers to avoid spending their money with companies that use these regressive tactics.

I think it’s important to distinguish types of barriers to entry. Some are “real” while others are “artificial”. For example, a real barrier to entry would be institutional knowledge about an industry while an artificial one would be an arbitrary TOS clause.

And disallowing scraping or making it difficult while refraining from providing an API for the same data is the arbitrary kind. The default state of the web is that it's trivially scrapable - you have to go out of your way to make it harder.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#223

Post by Xelbair »

akersten wrote: post #184
tgsovlerkhgsel wrote: post #156

Earlier quoted context omitted.

Not allowing the CFAA to be (ab)used to attempt to make scraping illegal makes sense. However, how is it reasonable to force a web site to serve its contents to a third-party company, without being allowed to make a decision whether to serve it or not? Serving the web site costs money, and the scraper surely isn't going to generate ad income...

Ugh, yeah, the more I think about this ruling, the less I like it. It's actually pretty insane to force a site to serve content. I think both parties are in the wrong here - HiQ for assuming they're entitled to receive a response from LinkedIn's webservers, and LinkedIn for abusing the CFAA to try to deny service rather than figure out a technical solution to their business problem. In my view: * The data is public,…

I would argue that under spirit of net neutrality you either serve your site to everyone equally(the public facing part) or to no one.

Hosting costs money, servers cost money.. but maybe create a public facing API that is way cheaper and easier to use than scraping your website? I see that ruling in positive light that it might promote more open and structured access to the public facing data.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#224

Post by TeMPOraL »

txcwpalpha wrote: post #69
admax88q wrote: post #67

Earlier quoted context omitted.

Lol really? I'm not "on" your site when I browse there. I asked your server to send me some data and it did so. Its real life equivalent to social engineering. Its so far not illegal for me to ask you things and for you to disclose them to me even if you weren't supposed to. I'm allowed to lie to you even to persuade you to tell me things.

You didn't "ask my server". You used a tool to extract data from my server. It's more akin to you standing just outside my property border and using a fishing pole to pull fish from a pond that is inside my property border. You're still trespassing even if your two feet aren't physically on my land. The common legal argument (see the second link in my above comment) is that accessing a web server actually does consti…

> You didn't "ask my server". You used a tool to extract data from my server.

You're always using a "tool" to "extract" data from a web server, unless you're manually operating a telnet session. A web browser is such tool, an incredibly complex and automated one. cURL is such a tool too, and so is cURL wrapped in a bash script. None of them go outside of what's allowed by HTTP protocol[0]. And the most core assumptions of the Internet and HTTP protocol combine into a simple rule: if it's a publicly routable server answering to HTTP requests, you can issue requests and receive whatever it sends. If a server wants to discriminate, it should set up an auth scheme.

--

[0] - protocol family at this point.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#225

Post by TeMPOraL »

mullingitover wrote: post #119
merb wrote: post #105

Earlier quoted context omitted.

jep. it's more like having a public store and only letting some people into it. like only males, no womans. because they clearly allowed the google bot.

More like only letting humans into it, and only one type of robot, the googlebot. Then this company is ignoring the posted rules and sending armies of robots into the store to photograph every square inch of the business.

Well I'm sorry, but if I opt to wrap a cURL call in a bash for loop, I'm still a human that tries to access the same resources, only with a different user agent.

There is legitimate individual interest in both scraping and non-browser HTTP sessions.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#226

Post by IfOnlyYouKnew »

tgsovlerkhgsel wrote: post #156
Animats wrote: post #22

This action does more than that. The court left the preliminary injunction against LinkedIn in place: "The district court granted hiQ’s motion. It ordered LinkedIn to withdraw its cease-and-desist letter, to remove any existing technical barriers to hiQ’s access to public profiles, and to refrain from putting in place any legal or technical measures with the effect of blocking hiQ’s access to public profiles." So Lin…

Not allowing the CFAA to be (ab)used to attempt to make scraping illegal makes sense. However, how is it reasonable to force a web site to serve its contents to a third-party company, without being allowed to make a decision whether to serve it or not? Serving the web site costs money, and the scraper surely isn't going to generate ad income...

You're inconsistent, and so are the courts and most comments here. Either you favour such conflicts to be decided by technological might, or by the clearly expressed will of the content publisher to have binding effect.

If you consider scrapers to have some sort of right to access any public website, any technological barriers inflict exactly the same harm as an injunction, assuming it is effective. IF you allow technical blocking, it would be preferable to allow blocking-by-clearly-stated-wish, because it would save everyone the costs of the arms race. It would also make both parties' success somewhat independent of the resources they can invest into outgunning their opponents.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#227

Post by dodobirdlord »

olau wrote: post #220
dodobirdlord wrote: post #204

Earlier quoted context omitted.

There's nothing hypocritical about it. Googlebot respects robots.txt configured on pages it scrapes. Google in turn expects that their own robots.txt will be respected. What's the issue? https://www.google.com/robots.txt

Can I politely point out that the conversation is not about respecting robots.txt. If you want to talk about this in terms of robots.txt, Google is thriving on the fact that other companies don't block their content in robots.txt, but at the same time Google blocks all of its content in its robots.txt.

> If you want to talk about this in terms of robots.txt, Google is thriving on the fact that other companies don't block their content in robots.txt, but at the same time Google blocks all of its content in its robots.txt.

It seems like you're stating this as though to cast some sort of moral aspersion. I don't get it. If other companies don't want Googlebot to scrape them they just have to say so. Most companies want Googlebot to scrape their content. Google doesn't want other people's scrapers to scrape Google's content. Nobody involved in any of this has done anything unreasonable or morally objectionable.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#228

Post by TeMPOraL »

dx87 wrote: post #34

Would that ruling mean that sites could no longer refuse to show content based on how they're accessed? For example, sites that won't load if the browser is in headless mode, or sites that depend on javascript as a way of blocking wget/curl.

I would very much hope so. These sites are a perversion of the idea of the Web, and only get away with it because fundamentals of the Web were created with friendly, cooperative users in mind - not with commercial users of today - so there's no enforcement of good behaviour.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#229

Post by 3xblah »

kabacha wrote: post #165
LeoPanthera wrote: post #158

Earlier quoted context omitted.

Does this prevent Google from returning captchas if you use a robot to scrape the search result pages, as they currently do?

I mean it should. That is also a huge anti-competitive action that just isn't pursued by anyone yet: google makes money of scraping and denies scrapers scraping them - that's all sorts of messed up. The problem is that someone would have to sue google first and no one will do that unless there's big business incentive and big business can already scrape the shit out of google. This is the weird thing about web-scrapi…

All true.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#230

Post by 3xblah »

ijidak wrote: post #148
penagwin wrote: post #57

Earlier quoted context omitted.

I feel like this is a really common theme I've seen several times. Something like "Music Lyric site X sues Google for embedding their lyrics in the results directly" which is funny because site X got the lyrics by scraping them from other sites. Plus Google only exists from scraping content, but I believe their TOS includes "don't scrape our content". I find it really funny that the scrapers are battling scrapers - l…

> Plus Google only exists from scraping content, but I believe their TOS includes "don't scrape our content". Yes. This is EXTREMELY frustrating. Of all companies to prevent scraping, Google is the most ironic. Especially since their goal is to organize the world's information, it shocks me that there's no way to get access to this organized information from machine to machine.

Perhaps this issue will be recognised in some of the antitrust investigations.

If I am not mistaken, they no longer claim "organize the world's information" as their goal.

Post reply on HN