Ask HN: Ethics and laws regarding scraping websites?

#1

Post by schtog » Sun, Jun 01, 2008, 4:01 PM UTC

What are the ethics and laws regarding scraping internet sites?

For one, we have robot.txt: http://en.wikipedia.org/wiki/Robot.txt

Is that the only thing that prevents(or prohibits) a robot/spider from scraping a site?

Can there be copyrighted material that is not allowed to be scraped?

I ask because I want to search/scrape a few hundred pages with similar content and present those results all the way to the buying of the product. I.e. not just like if you google you get a page that has the information. I want to present them with the options(maybe in a ranked order) where one click is needed to choose the product to buy, pretty much. Or one to search, one to choose and get to the productsite where you obviously have to do some more confirmation but you get the point...

Re: Ask HN: Ethics and laws regarding scraping websites?

#2

Post by puppetsock » Sun, Jun 01, 2008, 5:53 PM UTC

You just need to respect copyright. So if they explicitly allow you to scrape the data, or if you're operating within the "Fair Use" doctrine, then you're fine.

(IANAL)

Re: Ask HN: Ethics and laws regarding scraping websites?

#3

Post by dazzawazza » Sun, Jun 01, 2008, 5:58 PM UTC

I think this is a very grey area. If it's on the internet it's scrapable. Does that mean you can use it? You really need a lawyer.

Take for example Amazon reviews. They are there to be scraped but some of them are not available via the web service API due to copyright restrictions. That's a clear sign to me that they don't wouldn't want this information scraped. I'm not sure how a judge would see it though.

Another example is is licensed material. Take TV listings. They are listed all over the place but they are still under copyright so you can't just scrape them and use them on your site.

IANAL but my basic rule is that I scrape if I think I am going to offer the scraped sight something in return (usually traffic). So if it's mutually beneficial I feel OK. That doesn't mean it's legal though :(

Re: Ask HN: Ethics and laws regarding scraping websites?

#4

Post by nickb » Sun, Jun 01, 2008, 6:04 PM UTC

Depends on what and how you intend to use the scraped data for. I'd recommend you also check the TOS of the site you intend to scrape. Many sites explicitly say how their data can be used and how you should scrape it. Some sites also ban IPs that are scraping them (they assume it's a DoS type of an attack.

Re: Ask HN: Ethics and laws regarding scraping websites?

#5

Post by ntoshev » Sun, Jun 01, 2008, 6:06 PM UTC

If the websites you scrape don't want you to do so, they will update their robots.txt and ban you. If you don't respect that, they will ban your IP and/or take legal action.

So just try not to piss them off.

Re: Ask HN: Ethics and laws regarding scraping websites?

#6

Post by schtog » Sun, Jun 01, 2008, 6:10 PM UTC

but if we take google for an example. they give the serached sites something in return, they make the easy to find. but how does the google spider know that it can index the site? it checks for robots.txt obv but does it check for Copyright?

http://en.wikipedia.org/wiki/Fair_Use fair user seems to be only in the US, i potentially want to reahc the whole world even if the USA is a good place to start.

Re: Ask HN: Ethics and laws regarding scraping websites?

#7

Post by a-priori » Sun, Jun 01, 2008, 6:50 PM UTC

Ethically, I see no problem is doing any scraping as long as you credit the source and obey the robot.txt file. I figure, they were the ones who made the information freely available on the web.

Legally, however, it's a whole different beast that I'm not qualified to talk about.

Re: Ask HN: Ethics and laws regarding scraping websites?

#8

Post by tylercarbone » Sun, Jun 01, 2008, 6:57 PM UTC

So, copyright law doesn't care whether you scraped the data or acquired in some other way. That is, either you can use the information under copyright law, or you can't. If you can't use it, then getting it through some method other than scraping won't help you -- and if you can use it, then scraping won't change the legality.

So you need to make sure there isn't a copyright violation, which is going to depend on the specific information you're looking at.

There's still a potential problem, though, unrelated to copyright. After eBay v. Bidder's Edge, it can be trespass to chattels to scrape data in violation of a site's TOS. In the eBay case, the court held that it was in violation of trespass law because the eBay TOS prohibited robots... so it would have been fine if Bidder's Edge had taken the data manually.

Basically, you need to make sure the data isn't copyrighted, and you need to make sure that scraping the data isn't in violation of the site's TOS.

Re: Ask HN: Ethics and laws regarding scraping websites?

#9

Post by webwright » Sun, Jun 01, 2008, 7:45 PM UTC

Search engines all scrape... Titles, meta data, and some or all of the content.

I think a good rule of thumb is to consider whether the scraping target will benefit from being scraped. Most sites are delighted to be scraped by Google. Will your scraping drive sales/visitors to the target? Or will it cost sales/visitors? Will you link back (which helps them from an seo standpoint)?

Re: Ask HN: Ethics and laws regarding scraping websites?

#10

Post by xlnt » Sun, Jun 01, 2008, 8:31 PM UTC

nickb wrote: post #4

Depends on what and how you intend to use the scraped data for. I'd recommend you also check the TOS of the site you intend to scrape. Many sites explicitly say how their data can be used and how you should scrape it. Some sites also ban IPs that are scraping them (they assume it's a DoS type of an attack.

you should have a sleep between requests to avoid overloading their servers.

i don't know how much delay is best. does anyone else know?