Web Scraping to Create Open Data • Hacker News Board

Re: Web Scraping to Create Open Data

#11

Post by snurk » Wed, Mar 30, 2016, 6:54 PM UTC

maxaf wrote: post #4

rakoo wrote: post #3

"Web scraping to create Open Data" is the exact reason why weboob ( http://weboob.org/ ) was created and still thrives today. CityBikes already seems to be doing a big part of the job, and in Python nonetheless, so it should be easy to integrate its data and use it with Boobsize ( http://weboob.org/applications/boobsize.html )

That naming scheme definitely needs a long, hard rethink.

> ... long, hard ...

Heh.

Ok, but seriously, I look forward to seeing their appearance on r/drama when twitter discovers this.

Re: Web Scraping to Create Open Data

#12

Post by danvoell » Wed, Mar 30, 2016, 6:58 PM UTC

maxaf wrote: post #4

rakoo wrote: post #3

"Web scraping to create Open Data" is the exact reason why weboob ( http://weboob.org/ ) was created and still thrives today. CityBikes already seems to be doing a big part of the job, and in Python nonetheless, so it should be easy to integrate its data and use it with Boobsize ( http://weboob.org/applications/boobsize.html )

That naming scheme definitely needs a long, hard rethink.

agreed

Re: Web Scraping to Create Open Data

#13

Post by iolothebard » Wed, Mar 30, 2016, 7:16 PM UTC

toomuchtodo wrote: post #9

dsp1234 wrote: post #7

Earlier quoted context omitted.

In the US, there is no copyright protection for "facts" on their own. However, a compilation/database of facts can have copyright protections based on a 3 part test[0]. 1. the collection and assembly of pre-existing material, facts, or data; 2. the selection, coordination, or arrangement of those materials; and 3. the creation, by virtue of the particular selection, coordination, or arrangement of an original work of…

> So scraping the data, and rearranging the underlying facts into your own arrangement/organization is almost always not copyright infringement. I'm not so sure. It would definitely be illegal in the US for me to cherry pick data out of Google Maps and add it to OpenStreetMap (and OSM has policies addressing exactly this ).

Yet companies like LexisNexis get most their data they resell this way.

Re: Web Scraping to Create Open Data

#14

Post by kh_hk » Wed, Mar 30, 2016, 7:19 PM UTC

minimaxir wrote: post #10

kh_hk wrote: post #8

Earlier quoted context omitted.

> For the record, neither service did; in fact, both tried to recruit me as a result of the visualizations. The difference is that I am not using the data to create a direct competitor that could cause them to lose business. I do not understand that implication. How is providing bike share information creating a competitor? I can't run a bike sharing service.

Less a competitor, more a non-canonical source of information that they cannot manage. If your offshoot were to misrepresent data, for example, then you would become a liability, even if you weren't making money.

That's a fair point. Easily enforceable by a proper license. One example is ETALAB open data license.

Re: Web Scraping to Create Open Data

#15

Post by unsettledtck » Wed, Mar 30, 2016, 7:22 PM UTC

minimaxir wrote: post #5

I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down m…

Out of curiosity, where are the boundaries of your gray area when it comes to scraping?

Re: Web Scraping to Create Open Data

#16

Post by chrisweekly » Wed, Mar 30, 2016, 7:28 PM UTC

minimaxir wrote: post #10

kh_hk wrote: post #8

Earlier quoted context omitted.

> For the record, neither service did; in fact, both tried to recruit me as a result of the visualizations. The difference is that I am not using the data to create a direct competitor that could cause them to lose business. I do not understand that implication. How is providing bike share information creating a competitor? I can't run a bike sharing service.

Less a competitor, more a non-canonical source of information that they cannot manage. If your offshoot were to misrepresent data, for example, then you would become a liability, even if you weren't making money.

Random tangent: your comment about unmanaged non-canonical misrepresented data reminds me of Zillow, who publish "facts" about real estate transactions, with no mechanism for error correction. Afrw years ago they posted an erroneous sale of my house -- which transaction never took place -- listing a sale price 20% lower than we'd paid for it. It directly harmed me, when we later tried to sell the house, when potential buyers cited zillow's "estimates" which of course were artificially, drastically lower because of the phantom transaction. There was no avenue for recourse; angry tweets got a half-baked response from a junior social media person, but it was never resolved. I wonder how many others zillow must have messed up.

Re: Web Scraping to Create Open Data

#17

Post by l1n » Wed, Mar 30, 2016, 7:36 PM UTC

Heh. I do this with my Student Government data [1].

[1] https://umbc.lin.anticlack.com/finance/

Re: Web Scraping to Create Open Data

#18

Post by minimaxir » Wed, Mar 30, 2016, 7:39 PM UTC

unsettledtck wrote: post #15

minimaxir wrote: post #5

I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down m…

Out of curiosity, where are the boundaries of your gray area when it comes to scraping?

If the service has an API with a fair rate limit (Foursquare does at 5000 requests/hour), I believe that is ok, since that implies their architecture is built for massive data requests. On the other hand, bypassing those rate limits with proxies is definitely bad.

If a website does not have an API (BuzzFeed), I take care to only collect data that I need. Not anything that would damage the business. (E.g entire articles). Consequently, I sanitize the data of such things if I decide to release the dataset.

Re: Web Scraping to Create Open Data

#19

Post by rakoo » Wed, Mar 30, 2016, 7:55 PM UTC

maxaf wrote: post #4

rakoo wrote: post #3

"Web scraping to create Open Data" is the exact reason why weboob ( http://weboob.org/ ) was created and still thrives today. CityBikes already seems to be doing a big part of the job, and in Python nonetheless, so it should be easy to integrate its data and use it with Boobsize ( http://weboob.org/applications/boobsize.html )

That naming scheme definitely needs a long, hard rethink.

It's funny, everytime Weboob is presented somewhere, and everytime there is a post about the latest version of Weboob, the first comment is a variation of "it's sexist/boobs are unprofessional/grow up", and very very little time is spent talking about the actual thing, what it does and why its only goal is to become irrelevant. Sad thing.

Here's what they have to say about it, and why there's very little chance they will change anything:

http://laurent.bachelier.name/2013/12/weboob-the-asshole-det...

(This comment is not directed at you directly)

Re: Web Scraping to Create Open Data

#20

Post by maxaf » Wed, Mar 30, 2016, 8:01 PM UTC

rakoo wrote: post #19

maxaf wrote: post #4

Earlier quoted context omitted.

That naming scheme definitely needs a long, hard rethink.

It's funny, everytime Weboob is presented somewhere, and everytime there is a post about the latest version of Weboob, the first comment is a variation of "it's sexist/boobs are unprofessional/grow up", and very very little time is spent talking about the actual thing, what it does and why its only goal is to become irrelevant. Sad thing. Here's what they have to say about it, and why there's very little chance they…

It's a fun read. You know, I'm not normally one to jump on the "offended" bandwagon. In this case I felt compelled to speak up because the name makes it pretty much impossible to reference or recommend the project in a business setting. Wearing a t-shirt to work makes me feel like a true rebel; mentioning a project with components such as "boobsize" and "wetboobs" is going to simply be too weird for many people.