This is not legal advice, and it's not even a very good picture of the DMCA safe harbors, but hopefully it's enough to point you in the right direction. The Electronic Frontier Foundation has good resources on this, for example:
Ask HN: Ethics and laws regarding scraping websites?
11–20 of 25 posts
Re: Ask HN: Ethics and laws regarding scraping websites?
#12To reuse copyrighted content, you have to consider fair use eligibility and such -- but fortunately, not all data is locked down by copyright.
One of my favorite ten USSC rulings:
http://en.wikipedia.org/wiki/Feist_Publications_v._Rural_Tel...
""" It is a long-standing principle of United States copyright law that "information" is not copyrightable, O'Connor notes, but "collections" of information can be. Rural claimed a collection copyright in its directory. The court clarified that the intent of copyright law was not, as claimed by Rural and some lower courts, to reward the efforts of persons collecting information, but rather "to promote the Progress of Science and useful Arts" (U.S. Const. 1.8.8), that is, to encourage creative expression. Since facts are purely copied from the world around us, O'Connor concludes, "the sine qua non of copyright is originality". However, the standard for creativity is extremely low. It need not be novel, rather it only needs to possess a "spark" or "minimal degree" of creativity to be protected by copyright. ... In the late 1990s, Congress attempted to pass laws which would protect collections of data, but these measures failed. By contrast, the European Union has a sui generis (specific to that type of work) intellectual property protection for collections of data."""
Re: Ask HN: Ethics and laws regarding scraping websites?
#13I wrote a web crawler a few years back and at the time I didn't really understand the implications of having a crawler grab 20+ pages concurrently from a site.
I learned pretty fast when I found a few sites had banned my crawler.. Oops, sorry guys!
Re: Ask HN: Ethics and laws regarding scraping websites?
#14If the websites you scrape don't want you to do so, they will update their robots.txt and ban you. If you don't respect that, they will ban your IP and/or take legal action. So just try not to piss them off.
Re: Ask HN: Ethics and laws regarding scraping websites?
#15IANAL, but... To reuse copyrighted content, you have to consider fair use eligibility and such -- but fortunately, not all data is locked down by copyright. One of my favorite ten USSC rulings: http://en.wikipedia.org/wiki/Feist_Publications_v._Rural_Tel... """ It is a long-standing principle of United States copyright law that "information" is not copyrightable, O'Connor notes, but "collections" of information can b…
Re: Ask HN: Ethics and laws regarding scraping websites?
#16Depends on what and how you intend to use the scraped data for. I'd recommend you also check the TOS of the site you intend to scrape. Many sites explicitly say how their data can be used and how you should scrape it. Some sites also ban IPs that are scraping them (they assume it's a DoS type of an attack.
you should have a sleep between requests to avoid overloading their servers. i don't know how much delay is best. does anyone else know?
Re: Ask HN: Ethics and laws regarding scraping websites?
#17IANAL, but... To reuse copyrighted content, you have to consider fair use eligibility and such -- but fortunately, not all data is locked down by copyright. One of my favorite ten USSC rulings: http://en.wikipedia.org/wiki/Feist_Publications_v._Rural_Tel... """ It is a long-standing principle of United States copyright law that "information" is not copyrightable, O'Connor notes, but "collections" of information can b…
It is rumoured that some encyclopedias give biographies of ficticious people to enforce collective copyright. This wouldn't concern you if wanted information about a historical figure. However, it would be very problematic if you worked for a rival encyclopedia or wanted to make your own website of the encyclopedia.
Re: Ask HN: Ethics and laws regarding scraping websites?
#18Earlier quoted context omitted.
It is rumoured that some encyclopedias give biographies of ficticious people to enforce collective copyright. This wouldn't concern you if wanted information about a historical figure. However, it would be very problematic if you worked for a rival encyclopedia or wanted to make your own website of the encyclopedia.
Dictionaries have long done the same thing with regard to definitions. http://en.wikipedia.org/wiki/Copyright_trap
Re: Ask HN: Ethics and laws regarding scraping websites?
#19Here's an excerpt from the yellowpages.com TOS. Without it, their company names, addresses, and phone numbers would be more or less fair game for any competitor:
"You are prohibited from data mining, scraping, crawling, or using any process or processes that send automated queries to the YELLOWPAGES.COM Web site. You may not use the YELLOWPAGES.COM Web sites to compile a collection of listings, including a competing listing product or service."
Re: Ask HN: Ethics and laws regarding scraping websites?
#20Search engines all scrape... Titles, meta data, and some or all of the content. I think a good rule of thumb is to consider whether the scraping target will benefit from being scraped. Most sites are delighted to be scraped by Google. Will your scraping drive sales/visitors to the target? Or will it cost sales/visitors? Will you link back (which helps them from an seo standpoint)?
They scrape if the owners have indicated they wish their site to be scraped by not forbidding access via the robots.txt file.