Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

cdn.ca9.uscourts.gov

Post reply on HN
Print view View original

271–280 of 293 posts

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#271

Post by dx034 »

devmunchies wrote: post #173
mullingitover wrote: post #2

> LinkedIn has taken steps to protect the data on its website from what it perceives as misuse or misappropriation. The instructions in LinkedIn’s “robots.txt” file—a text file used by website owners to communicate with search engine crawlers and other web robots—prohibit access to LinkedIn servers via automated bots, except that certain entities, like the Google search engine, have express permission from LinkedIn f…

>except that certain entities, like the Google search engine, have express permission from LinkedIn for bot access how does this work technically? i just tried crawling a friend's profile using curl and set my user agent to Google's bot and it still was blocked.

Google describes how to verify Googlebot here: https://support.google.com/webmasters/answer/80553?hl=en

Most other search engine crawlers provide similar methods.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#272

Post by sezna »

ddingus wrote: post #183
sezna wrote: post #172

Earlier quoted context omitted.

It makes sense and it is how the internet works. Servers cherry pick who sees their content all the time. Scrapers are often blocked, as are entire IP address ranges. Things like Selenium server scrapers can be (approximately) detected and often are denied access. I’m not sure about being anti-competitive. Serving a website is an action in which you open up your resources for others to access. My friend runs an open…

He should open up a Patreon, tip jar, something to get that funded. Could also delay results, offer reduced temporal precision and other things to differentiate use cases.

He and I both have similar free open source websites with donate buttons. They are rarely clicked. Ad revenue over a month for me has been ~$400 while donations over two years have totaled $20. There are about 80,000 unique visitors per month.

It is nice to think donation platforms can fund high traffic open source projects, but this is simply not the case.

In any regard, I fear the potential of this ruling limiting developers’ ability to protect their servers and making us all roll over to the big players with their hefty scrapers taking all of our data for resale.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#273

Post by prirun »

eBay had better lawyers than LinkedIn:

https://casetext.com/case/ebay-v-bidders-edge

I'm glad this court ruled it wasn't a violation of CFAA. But using trespass to prevent it seems reasonable. A private business should be allowed to restrict certain kinds of use of its resources (servers, bandwidth, etc), especially if it is beyond typical use. But if the load is typical and doesn't actually harm LinkedIn, it seems less reasonable to restrict them. If LinkedIn doesn't want automated access to their data because it is too much of a load on their servers, then they should be required to ban ALL automated access, including Google's bots. Of course they want Google's bots because that sends them traffic.

Another reason I think it was stupid for LinkedIn to use CFAA is that it sets them up to be a protected computer system, with protected information. If that is the case, it seems they could be liable for disclosing the information to someone a user didn't want, like a stalked. It's rather dumb: LI is claiming they host protected information, but it is only protected against someone that might compete with them.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#274

Post by UncleMeat »

hitekker wrote: post #143
UncleMeat wrote: post #99

Earlier quoted context omitted.

A friend of mine from grad school was very involved in legal issues related to cfaa stuff. According to him, weev really got screwed because he failed "the punk test", which discouraged lawyers from wanting to use him as a test case.

Curious question: what is "the punk test"?

"Is this person going to say idiotic nonsense that tanks the case because the judge or jury thinks they are just awful?"

Weev may have been a good test case if he wasn't a white supremacist.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#275

Post by test6554 »

tgsovlerkhgsel wrote: post #156
Animats wrote: post #22

This action does more than that. The court left the preliminary injunction against LinkedIn in place: "The district court granted hiQ’s motion. It ordered LinkedIn to withdraw its cease-and-desist letter, to remove any existing technical barriers to hiQ’s access to public profiles, and to refrain from putting in place any legal or technical measures with the effect of blocking hiQ’s access to public profiles." So Lin…

Not allowing the CFAA to be (ab)used to attempt to make scraping illegal makes sense. However, how is it reasonable to force a web site to serve its contents to a third-party company, without being allowed to make a decision whether to serve it or not? Serving the web site costs money, and the scraper surely isn't going to generate ad income...

This second part is pretty stupid, however, now that we are at this point, Linkedin still has the ability to decide which of its information is public and which is not. By making all of its information private, it can take back control.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#276

Post by anticensor »

fauigerzigerk wrote: post #270
anticensor wrote: post #265

Earlier quoted context omitted.

Europe has database rights, which has a fair dealing exemption for data analysis.

I'm not sure what "database rights" refers to specifically, but the whole matter is actually rather complicated, because the EU copyright directive has a lot of optional exceptions that member states may or may not adopt. Most of these exceptions only apply to non-commercial use though. So they wouldn't apply in a case like hiQ. UK specific exceptions are explained here: https://www.gov.uk/guidance/exceptions-to-copy…

Database rights are a copyright-like intellectual property regime for databases.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#277

Post by silentguy »

prirun wrote: post #273

eBay had better lawyers than LinkedIn: https://casetext.com/case/ebay-v-bidders-edge I'm glad this court ruled it wasn't a violation of CFAA. But using trespass to prevent it seems reasonable. A private business should be allowed to restrict certain kinds of use of its resources (servers, bandwidth, etc), especially if it is beyond typical use. But if the load is typical and doesn't actually harm LinkedIn, it seems l…

> If LinkedIn doesn't want automated access to their data because it is too much of a load on their servers, then they should be required to ban ALL automated access, including Google's bots. Of course they want Google's bots because that sends them traffic.

By that logic, I should have access to linked premium features for free. Why should linkedin give more data to people who pay?

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#278

Post by ehvatum »

RobAley wrote: post #264
eru wrote: post #246

Earlier quoted context omitted.

Why should you be forced to serve content to people who won't look at your ads?

Like disabled users with screen-readers?

I suppose we can give them a pass if they solve a bunch of captchas.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#279

Post by hgoel »

akersten wrote: post #184
tgsovlerkhgsel wrote: post #156

Earlier quoted context omitted.

Not allowing the CFAA to be (ab)used to attempt to make scraping illegal makes sense. However, how is it reasonable to force a web site to serve its contents to a third-party company, without being allowed to make a decision whether to serve it or not? Serving the web site costs money, and the scraper surely isn't going to generate ad income...

Ugh, yeah, the more I think about this ruling, the less I like it. It's actually pretty insane to force a site to serve content. I think both parties are in the wrong here - HiQ for assuming they're entitled to receive a response from LinkedIn's webservers, and LinkedIn for abusing the CFAA to try to deny service rather than figure out a technical solution to their business problem. In my view: * The data is public,…

The ruling doesn't seem to say that they can't throttle access atleast.

Re: 9th Circuit holds that scraping a public website does not violate the CFAA [pdf]

#280

Post by joncrane »

lwf wrote: post #160
mullingitover wrote: post #2

> LinkedIn has taken steps to protect the data on its website from what it perceives as misuse or misappropriation. The instructions in LinkedIn’s “robots.txt” file—a text file used by website owners to communicate with search engine crawlers and other web robots—prohibit access to LinkedIn servers via automated bots, except that certain entities, like the Google search engine, have express permission from LinkedIn f…

Even if the case was tried today, 9th Cir. isn't binding on other regions of the US, and there's a bit of a split, as detailed in the opinion[1]: > In recognizing that the CFAA is best understood as an anti-intrusion statute and not as a “misappropriation statute,” we rejected the contract-based interpretation of the CFAA’s “without authorization” provision adopted by some of our sister circuits. Compare Facebook, In…

Is there an AWS region in the District governed by this case, so you can just do all your web scraping from instances in that region?
Post reply on HN