We scraped the World Bank's website
cgdev.org
We scraped the World Bank's website
1–10 of 37 posts
Re: We scraped the World Bank's website
#2I'm not sure what to make of this. On the one hand, the data could be useful.
On the other hand, the data was probably already useful through the world bank's tool. Furthermore, it's reasonable to assume that the data was only released because the people who had to make the final decision were convinced that the effort required to reassemble the data would be prohibitive. The fact that it's now all been released might have a chilling effect on future releases.
Re: We scraped the World Bank's website
#3I wonder if they would have released this if Weev had not been freed.
Re: We scraped the World Bank's website
#4There is an appendix to the paper describing how to install and run the author's script. I'd like to take a look at the source, but can't find an actual link to the code. Am I overlooking something?
Re: We scraped the World Bank's website
#5An interesting and somewhat related (but only tangentially) article I read a couple of days ago found that nearly 1/3rd of World Bank reports are never read, not even by a single person: http://www.washingtonpost.com/blogs/wonkblog/wp/2014/05/08/t...
It was submitted to HN (https://news.ycombinator.com/item?id=7715881) by another user, but probably never got traction because the title of the article is very vague.
Re: We scraped the World Bank's website
#6I'm not sure what to make of this. On the one hand, the data could be useful. On the other hand, the data was probably already useful through the world bank's tool. Furthermore, it's reasonable to assume that the data was only released because the people who had to make the final decision were convinced that the effort required to reassemble the data would be prohibitive. The fact that it's now all been released migh…
Not sure I follow. Do you think the World Bank will now hesitate to release their data because someone scraped it?
They have an open data policy, using CC-BY [1], so unless this scraping effort took data that wasn't covered by that it should be ok I think.
[1] http://web.worldbank.org/WBSITE/EXTERNAL/NEWS/0,,contentMDK:...
Re: We scraped the World Bank's website
#7There is an appendix to the paper describing how to install and run the author's script. I'd like to take a look at the source, but can't find an actual link to the code. Am I overlooking something?
1. go here http://www.cgdev.org/section/publications?f[0]=field_documen...
2. click on the link to "We Just Ran Twenty-Three Million Queries of the World Bank's Web Site"
3. click on the "Data & Analysis" tab
4. scroll to the bottom and there are download links to harvester_parameters.py, harvester.py and unloader.py
BUT i can't seem to actually download them, as there is some redirect that fails :( i tried creating an account on that site as well. anyone else have any luck?
Re: We scraped the World Bank's website
#8There is an appendix to the paper describing how to install and run the author's script. I'd like to take a look at the source, but can't find an actual link to the code. Am I overlooking something?
Welcome to most software with academic papers published about it.
It's pretty typical to find papers written about a piece of software or a software technique with impressive claims and the software is totally unavailable to the larger academic community (neither in binaries or source).
Re: We scraped the World Bank's website
#9i thought the point of this way, gee, the world bank makes it hard for you to get data out, so we scraped them, and here is the data for you guys to play around with. I looked for a bit, but couldnt find the data -- am I missing something?
Re: We scraped the World Bank's website
#10There is an appendix to the paper describing how to install and run the author's script. I'd like to take a look at the source, but can't find an actual link to the code. Am I overlooking something?
1. go here http://www.cgdev.org/section/publications?f[0]=field_documen... 2. click on the link to "We Just Ran Twenty-Three Million Queries of the World Bank's Web Site" 3. click on the "Data & Analysis" tab 4. scroll to the bottom and there are download links to harvester_parameters.py, harvester.py and unloader.py BUT i can't seem to actually download them, as there is some redirect that fails :( i tried creating…
[deleted]