Live data from Hacker News

Ask HN: Legality of Web Scraping User-Submitted Content?

news.ycombinator.com

1–10 of 48 posts

Ask HN: Legality of Web Scraping User-Submitted Content?

#1
Is it legal to scrape another website for user submitted content? For example, if there was a news site that got its news solely from users submitting their own personally-written articles and stories -- who owns that content? Does the end user relinquish ownership once they submit that content?

Is it legal for another company to "scrape" that content and use it on their site, only removing it if the user who submitted it in the first place asks them to, either directly or via legal means?

Thoughts and insight would be appreciated!

Re: Ask HN: Legality of Web Scraping User-Submitted Content?

#2
Well, there's the real world answer, and the legal answer. Not too sure about the legal answer, thats always murky. Some things are copyrighted, some aren't. You can't just take someone's story, or art, and use it on your own site without permission. Usually the creator has all rights until they are given away. Even if they have uploaded it to another site, that doesn't mean they have given you permission though. On the other hand, some information can't be controlled. If Bob says it's 70 degrees in San Francisco right now, you can totally say its 70 degrees. If the Mets won, you're welcome to say the Mets won. The Drudge Report does nothing except report headlines.

In the real world, however, people steal information all the time. Its not polite though. Usually people ask for attributes. I don't think a policy of only removing it when someone asks you to would be polite. That's like stealing something when no-one is home and leaving a note saying you'll return it if they ask you to.

Also, pure scraping, even of non-copyrighted information can get you into trouble if the other person had paid for that information, like a news site. They pay for their news. Scraping it and making your own news site (with full content, not just headlines) is illegal.

So the short answer is that the original creator still owns the content, and no you probably can't have it.

Re: Ask HN: Legality of Web Scraping User-Submitted Content?

#3
post #2

Well, there's the real world answer, and the legal answer. Not too sure about the legal answer, thats always murky. Some things are copyrighted, some aren't. You can't just take someone's story, or art, and use it on your own site without permission. Usually the creator has all rights until they are given away. Even if they have uploaded it to another site, that doesn't mean they have given you permission though. On…

I guess my followup with that would be this:

What if the user is submitting content that they would very clearly want re-distributed? I guess my question with that is, if you take the end-user who is submitting the content out of the equation, does the site that is being scraped from have any kind of leg to stand on if they don't want you scraping their content (excluding technical means that they might implement), assuming that the site being scraped from does not force their end-user who is submitting the content to agree that the site that they are submitting the content to "owns" the content, once it is submitted?

Re: Ask HN: Legality of Web Scraping User-Submitted Content?

#4
A few years back I wanted to set up a menu service - putting a bunch of restaurant menus online so that others could search. My lawyer advised that because the restaurant made these materials open in the public domain, we could basically do whatever we wanted with them. As if they were public property.

I'm not sure how this relates to other kind of content from a legal standpoint, but I've used it to ask myself 'did this person intend for this information to be public' as sort of an ethical guideline.

Re: Ask HN: Legality of Web Scraping User-Submitted Content?

#5
In the legal world, whenever text is set in fixed form, it is automatically copyrighted by its author, whether they claim it or not, unless they specifically say otherwise. "Specifically say otherwise" probably includes anything they might have agreed to in the TOS for a particular site, although the legal waters there are largely untested. The only exemption to this auto-copyright is statistics, phone numbers, or other non-copyrightable content (see CBC vs. MLB).

IANAL, but depending on the nature of your service and specifically how the content is collected, you may qualify for DMCA safe harbor protections. This means that if you remove things in a timely manner upon request nobody can sue you. This is how Google caches the whole internet without getting sued.

That's all legal mumbo-jumbo. The real world answer is that some people will get mad regardless of the law, so take their content down and apologize. Follow robots.txt guidelines. Don't post takedown replys a la PirateBay. Generally act sane. If you do all of the above you'll probably be ok.

Re: Ask HN: Legality of Web Scraping User-Submitted Content?

#6
post #4

A few years back I wanted to set up a menu service - putting a bunch of restaurant menus online so that others could search. My lawyer advised that because the restaurant made these materials open in the public domain, we could basically do whatever we wanted with them. As if they were public property. I'm not sure how this relates to other kind of content from a legal standpoint, but I've used it to ask myself 'did…

Woah, that is seriously not correct advice. Was your lawyer even a copyright lawyer?

Re: Ask HN: Legality of Web Scraping User-Submitted Content?

#7
Check to see if the site that you're considering scraping has any sort of "terms and conditions", "terms of use", "legals", etc page. If it does, read that page in detail. If it doesn't, ask the site for permission to scrape.

If the site doesn't have one of those pages and you don't ask for permission, you're not only putting yourself in danger of legal action, but you're also depending on a data source that isn't reliable.

Remember, it'd only be a matter of time before they notice you scraping, and take measures to stop it.

Re: Ask HN: Legality of Web Scraping User-Submitted Content?

#8
I will suggest that you mention clearly the origin of anything you get from anywhere on the internet.

The truth is, it’s hard to find a site without the mention of copyright in terms and conditions page.

Overall it will depend of the use you make of the copy content. Trouble starts when you use someone’s material in a line of business that competes with the owners.

Re: Ask HN: Legality of Web Scraping User-Submitted Content?

#9
Technically, you own whatever you write. So I own the copyright on this message, unless I granted the copyright to YC News when I signed up (I don't remember). Assuming that I did not, then I retain copyright, and you have to get the ok from me, individually.

That's the theory anyway. Talk to a lawyer.

Post reply on HN