Live data from Hacker News

Web Scraping to Create Open Data

blog.scrapinghub.com

31–40 of 58 posts

Re: Web Scraping to Create Open Data

#31
post #4
post #3

"Web scraping to create Open Data" is the exact reason why weboob ( http://weboob.org/ ) was created and still thrives today. CityBikes already seems to be doing a big part of the job, and in Python nonetheless, so it should be easy to integrate its data and use it with Boobsize ( http://weboob.org/applications/boobsize.html )

That naming scheme definitely needs a long, hard rethink.

Are you referring to weBOOB.com or scRAPINGhub.com or both?

Re: Web Scraping to Create Open Data

#32
post #19
post #4

Earlier quoted context omitted.

That naming scheme definitely needs a long, hard rethink.

It's funny, everytime Weboob is presented somewhere, and everytime there is a post about the latest version of Weboob, the first comment is a variation of "it's sexist/boobs are unprofessional/grow up", and very very little time is spent talking about the actual thing, what it does and why its only goal is to become irrelevant. Sad thing. Here's what they have to say about it, and why there's very little chance they…

I clicked the link and got some "hipsters not allowed" page. Not exactly helping the point that it's not just childish.

Re: Web Scraping to Create Open Data

#33
post #4
post #3

"Web scraping to create Open Data" is the exact reason why weboob ( http://weboob.org/ ) was created and still thrives today. CityBikes already seems to be doing a big part of the job, and in Python nonetheless, so it should be easy to integrate its data and use it with Boobsize ( http://weboob.org/applications/boobsize.html )

That naming scheme definitely needs a long, hard rethink.

A long, hard rethink?

Phrasing!

Re: Web Scraping to Create Open Data

#34
post #19
post #4

Earlier quoted context omitted.

That naming scheme definitely needs a long, hard rethink.

It's funny, everytime Weboob is presented somewhere, and everytime there is a post about the latest version of Weboob, the first comment is a variation of "it's sexist/boobs are unprofessional/grow up", and very very little time is spent talking about the actual thing, what it does and why its only goal is to become irrelevant. Sad thing. Here's what they have to say about it, and why there's very little chance they…

> It's funny, everytime Weboob is presented somewhere, and everytime there is a post about the latest version of Weboob, the first comment is a variation of "it's sexist/boobs are unprofessional/grow up", and very very little time is spent talking about the actual thing, what it does and why its only goal is to become irrelevant. Sad thing.

That in itself is a very good reason why the naming scheme needs a long, hard rethink. It's distracting.

Put another way, I wouldn't blame everyone else if every time my hypothetical company "Natzie" was mentioned the conversation devolved into something unrelated to the reason it was mentioned. If you want to succeed, you need to make decisions based on the real world, not how the world should be (not that you can't push limits, but baby steps are often needed).

Re: Web Scraping to Create Open Data

#35
post #4
post #3

"Web scraping to create Open Data" is the exact reason why weboob ( http://weboob.org/ ) was created and still thrives today. CityBikes already seems to be doing a big part of the job, and in Python nonetheless, so it should be easy to integrate its data and use it with Boobsize ( http://weboob.org/applications/boobsize.html )

That naming scheme definitely needs a long, hard rethink.

On their homepage you see "anal+" image....

Re: Web Scraping to Create Open Data

#36
post #32
post #19

Earlier quoted context omitted.

It's funny, everytime Weboob is presented somewhere, and everytime there is a post about the latest version of Weboob, the first comment is a variation of "it's sexist/boobs are unprofessional/grow up", and very very little time is spent talking about the actual thing, what it does and why its only goal is to become irrelevant. Sad thing. Here's what they have to say about it, and why there's very little chance they…

I clicked the link and got some "hipsters not allowed" page. Not exactly helping the point that it's not just childish.

"weboob the asshole detector"

I think it probably /is/ an asshole detector, but not quite in the way that the authors of that webpage /think/ it is.

from Wikipedia:

Psychological projection is a theory in psychology in which humans defend themselves against their own unpleasant impulses by denying their existence while attributing them to others.

Re: Web Scraping to Create Open Data

#37
post #19

Earlier quoted context omitted.

It's funny, everytime Weboob is presented somewhere, and everytime there is a post about the latest version of Weboob, the first comment is a variation of "it's sexist/boobs are unprofessional/grow up", and very very little time is spent talking about the actual thing, what it does and why its only goal is to become irrelevant. Sad thing. Here's what they have to say about it, and why there's very little chance they…

Don't click the link above. It may crash your browser. ------------------------------ Full text: We often get complaints around Weboob‘s name, and the various application names. There’s no denying they’re childish. What they are not, however, is sexist. There is “boob” in the main name, and “boob” is a friendly name referring to (mostly female) breasts. We would, for example, avoid using “tits” or “cunt”, because the…

It may crash your browser >intentionally<

Re: Web Scraping to Create Open Data

#38
post #7

Earlier quoted context omitted.

In the US, there is no copyright protection for "facts" on their own. However, a compilation/database of facts can have copyright protections based on a 3 part test[0]. 1. the collection and assembly of pre-existing material, facts, or data; 2. the selection, coordination, or arrangement of those materials; and 3. the creation, by virtue of the particular selection, coordination, or arrangement of an original work of…

> So scraping the data, and rearranging the underlying facts into your own arrangement/organization is almost always not copyright infringement. I'm not so sure. It would definitely be illegal in the US for me to cherry pick data out of Google Maps and add it to OpenStreetMap (and OSM has policies addressing exactly this ).

What part of the law does this fall under? Do people get arrested for this? (i.e. criminal) What's the worst that can happen?

Re: Web Scraping to Create Open Data

#39
post #19
post #4

Earlier quoted context omitted.

That naming scheme definitely needs a long, hard rethink.

It's funny, everytime Weboob is presented somewhere, and everytime there is a post about the latest version of Weboob, the first comment is a variation of "it's sexist/boobs are unprofessional/grow up", and very very little time is spent talking about the actual thing, what it does and why its only goal is to become irrelevant. Sad thing. Here's what they have to say about it, and why there's very little chance they…

Warning, this link either goes to CrashSafari or http://no-hipsters-allowed.t28.net/ .

Re: Web Scraping to Create Open Data

#40
post #6

I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down m…

Disclaimer, I wrote the article. > I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. It was not my intention to give that implication. The main implication behind CityBikes is that public services should already provide this information since, well, it…

Why does your blog intentionally crash browsers that it thinks are Safari?
Post reply on HN