Heh. I do this with my Student Government data [1]. [1] https://umbc.lin.anticlack.com/finance/
You need to fix the certificate before showing that to the public.
Web Scraping to Create Open Data
41–50 of 58 posts
Re: Web Scraping to Create Open Data
#42Earlier quoted context omitted.
> So scraping the data, and rearranging the underlying facts into your own arrangement/organization is almost always not copyright infringement. I'm not so sure. It would definitely be illegal in the US for me to cherry pick data out of Google Maps and add it to OpenStreetMap (and OSM has policies addressing exactly this ).
What part of the law does this fall under? Do people get arrested for this? (i.e. criminal) What's the worst that can happen?
Re: Web Scraping to Create Open Data
#43I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down m…
Why do you think scraping needs justifying in the first place?
Re: Web Scraping to Create Open Data
#44"Web scraping to create Open Data" is the exact reason why weboob ( http://weboob.org/ ) was created and still thrives today. CityBikes already seems to be doing a big part of the job, and in Python nonetheless, so it should be easy to integrate its data and use it with Boobsize ( http://weboob.org/applications/boobsize.html )
This script [1] using pybikes would accomplish a similar thing.
Only problem with using pybikes natively are some systems we call asynchronous. These are bike share websites that require actually clicking on a station to get the status information. This means that for getting accurate information on the entire the feed (let's say 500 stations), you would have to go through each of them. On these cases, it's way easier to just use the API.
[1]: https://gist.github.com/eskerda/bbd65539048a53eadfccc5d535ad...
Re: Web Scraping to Create Open Data
#45Earlier quoted context omitted.
What part of the law does this fall under? Do people get arrested for this? (i.e. criminal) What's the worst that can happen?
https://en.wikipedia.org/wiki/Copyright_infringement http://www.copyright.gov/title17/92chap5.html#501 https://www.law.cornell.edu/uscode/text/17/chapter-5 https://www.lib.purdue.edu/uco/CopyrightBasics/penalties.htm...
That's begging the question of whether Google's data on public streets is actually protected by copyright under U.S. law.
Re: Web Scraping to Create Open Data
#46Earlier quoted context omitted.
It's funny, everytime Weboob is presented somewhere, and everytime there is a post about the latest version of Weboob, the first comment is a variation of "it's sexist/boobs are unprofessional/grow up", and very very little time is spent talking about the actual thing, what it does and why its only goal is to become irrelevant. Sad thing. Here's what they have to say about it, and why there's very little chance they…
Interesting — that link is either down, or it's set up to present several different troll responses depending on the http referrer. When I click, I'm redirected to a simple image. When I request it in incognito mode, I'm redirected to a "crash safari" url.
No problem here. RefControl set to block by default.
Re: Web Scraping to Create Open Data
#47Earlier quoted context omitted.
Disclaimer, I wrote the article. > I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. It was not my intention to give that implication. The main implication behind CityBikes is that public services should already provide this information since, well, it…
Why does your blog intentionally crash browsers that it thinks are Safari?
What do you mean by intentionally?
Re: Web Scraping to Create Open Data
#48Earlier quoted context omitted.
Disclaimer, I wrote the article. > I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. It was not my intention to give that implication. The main implication behind CityBikes is that public services should already provide this information since, well, it…
Why does your blog intentionally crash browsers that it thinks are Safari?
[deleted]
Re: Web Scraping to Create Open Data
#49Earlier quoted context omitted.
Disclaimer, I wrote the article. > I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. It was not my intention to give that implication. The main implication behind CityBikes is that public services should already provide this information since, well, it…
Why does your blog intentionally crash browsers that it thinks are Safari?
You appear to have replied to the wrong comment. ScrapingHub is not the site that attempts to crash Safari - that is weboob.
Re: Web Scraping to Create Open Data
#50So what is the legality of this? Apart from the risk of having someone pull the plug on the way one takes the information out, when is something without a proper license able to be used?
IANAL but in the EU at least, even databases comprised of simple "facts" are protected. It's a sad state of affairs when i'm not even allowed to scrape data generated using taxpayers' money, like the (required by EU laws) noise maps for cities, which I'd like to use to augment real estate offers, for example.
"Europe" would like to partially fund that noise database with income from businesses that use it. The result is less taxpayer money us needed.
I think it's only the UK that has copyrightable fact databases