Web Scraping to Create Open Data
blog.scrapinghub.com
Web Scraping to Create Open Data
1–10 of 58 posts
Re: Web Scraping to Create Open Data
#2So what is the legality of this? Apart from the risk of having someone pull the plug on the way one takes the information out, when is something without a proper license able to be used?
Re: Web Scraping to Create Open Data
#3"Web scraping to create Open Data" is the exact reason why weboob (http://weboob.org/) was created and still thrives today. CityBikes already seems to be doing a big part of the job, and in Python nonetheless, so it should be easy to integrate its data and use it with Boobsize (http://weboob.org/applications/boobsize.html)
Re: Web Scraping to Create Open Data
#4"Web scraping to create Open Data" is the exact reason why weboob ( http://weboob.org/ ) was created and still thrives today. CityBikes already seems to be doing a big part of the job, and in Python nonetheless, so it should be easy to integrate its data and use it with Boobsize ( http://weboob.org/applications/boobsize.html )
That naming scheme definitely needs a long, hard rethink.
Re: Web Scraping to Create Open Data
#5I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do.
I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down my results, I would not contest, since data is what drives the Internet ecosystem.
(For the record, neither service did; in fact, both tried to recruit me as a result of the visualizations. The difference is that I am not using the data to create a direct competitor that could cause them to lose business.)
Re: Web Scraping to Create Open Data
#6I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down m…
Disclaimer, I wrote the article.
> I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do.
It was not my intention to give that implication. The main implication behind CityBikes is that public services should already provide this information since, well, it is a public service. On the same line, a private company providing a public service should already do so. See motives [1].
> I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down my results, I would not contest, since data is what drives the Internet ecosystem.
Same as CityBikes is doing. If we receive a cease and desist, we remove their service from our API. As for Foursquare, I do not see Foursquare as a public service. Your taxdollars at work, and all that.
I tried to keep the article balanced but maybe it wasn't clear. There are many transportation companies willing and happy to be scraped, or looking forward to provide their information for people to reuse [2].
[1]: https://blog.scrapinghub.com/2016/03/30/web-scraping-to-crea...
Re: Web Scraping to Create Open Data
#7So what is the legality of this? Apart from the risk of having someone pull the plug on the way one takes the information out, when is something without a proper license able to be used?
In the US, there is no copyright protection for "facts" on their own. However, a compilation/database of facts can have copyright protections based on a 3 part test[0].
1. the collection and assembly of pre-existing material, facts, or data;
2. the selection, coordination, or arrangement of those materials; and
3. the creation, by virtue of the particular selection, coordination, or arrangement of an original work of authorship.
However, if what you're scraping are not "facts", but some creative works, such as blog posts, product descriptions, etc, then it is likely to be copyright infringement.
Then on top of that, even if there is copyright infringement, other defenses such as a license to use the data, or fair use may apply.
Re: Web Scraping to Create Open Data
#8I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down m…
> For the record, neither service did; in fact, both tried to recruit me as a result of the visualizations. The difference is that I am not using the data to create a direct competitor that could cause them to lose business.
I do not understand that implication. How is providing bike share information creating a competitor? I can't run a bike sharing service.
Re: Web Scraping to Create Open Data
#9So what is the legality of this? Apart from the risk of having someone pull the plug on the way one takes the information out, when is something without a proper license able to be used?
In the US, there is no copyright protection for "facts" on their own. However, a compilation/database of facts can have copyright protections based on a 3 part test[0]. 1. the collection and assembly of pre-existing material, facts, or data; 2. the selection, coordination, or arrangement of those materials; and 3. the creation, by virtue of the particular selection, coordination, or arrangement of an original work of…
> So scraping the data, and rearranging the underlying facts into your own arrangement/organization is almost always not copyright infringement.
I'm not so sure. It would definitely be illegal in the US for me to cherry pick data out of Google Maps and add it to OpenStreetMap (and OSM has policies addressing exactly this).
Re: Web Scraping to Create Open Data
#10I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down m…
> For the record, neither service did; in fact, both tried to recruit me as a result of the visualizations. The difference is that I am not using the data to create a direct competitor that could cause them to lose business. I do not understand that implication. How is providing bike share information creating a competitor? I can't run a bike sharing service.
Less a competitor, more a non-canonical source of information that they cannot manage.
If your offshoot were to misrepresent data, for example, then you would become a liability, even if you weren't making money.