Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We scraped the World Bank's website

cgdev.org

Post reply on HN
Print view View original

11–20 of 37 posts

Re: We scraped the World Bank's website

#11

Post by ohashi »

mileswu wrote: post #5

An interesting and somewhat related (but only tangentially) article I read a couple of days ago found that nearly 1/3rd of World Bank reports are never read, not even by a single person: http://www.washingtonpost.com/blogs/wonkblog/wp/2014/05/08/t... It was submitted to HN ( https://news.ycombinator.com/item?id=7715881 ) by another user, but probably never got traction because the title of the article is very vague.

It sounds like a marketing problem. Although, I suspect the PDFs aren't the only dissemination method (as stated in the article). After someone spends so much time writing a report, it probably sits in their consciousness and spreads via the person's other interactions (eg shaping their perspective, thinking, and pursuits). With so many people producing so much content on a daily basis, it's hard to imagine people actually reading it all. So hopefully the good ideas get kept in the person's thoughts and come out repeatedly until it's heard. Otherwise, I am not sure there really is a good filtering mechanism on a system scale.

Re: We scraped the World Bank's website

#12

Post by obstacle1 »

officialjunk wrote: post #7
obstacle1 wrote: post #4

There is an appendix to the paper describing how to install and run the author's script. I'd like to take a look at the source, but can't find an actual link to the code. Am I overlooking something?

1. go here http://www.cgdev.org/section/publications?f[0]=field_documen... 2. click on the link to "We Just Ran Twenty-Three Million Queries of the World Bank's Web Site" 3. click on the "Data & Analysis" tab 4. scroll to the bottom and there are download links to harvester_parameters.py, harvester.py and unloader.py BUT i can't seem to actually download them, as there is some redirect that fails :( i tried creating…

Perfect, thanks.

I got the redirect error too when I tried to download _all_ files. However when I deselected all and just checked the readme, csv data, and python scripts, it worked just fine.

Re: We scraped the World Bank's website

#13

Post by yp_all »

"public domain... but can [only] be accessed in small pieces"

Sounds like the wonderful world of "APIs" on the www.

This sort of data should be on an FTP server.

I can build my own "apps". Give me the option of raw data.

Just my opinion, nothing more.

Re: We scraped the World Bank's website

#14

Post by bybjorn »

dalek2point3 wrote: post #9

i thought the point of this way, gee, the world bank makes it hard for you to get data out, so we scraped them, and here is the data for you guys to play around with. I looked for a bit, but couldnt find the data -- am I missing something?

The last sentence on that page reads: "The full data can be downloaded at www.cgdev.org/povcalnet."

Re: We scraped the World Bank's website

#16

Post by aaron-lebo »

yp_all wrote: post #13

"public domain... but can [only] be accessed in small pieces" Sounds like the wonderful world of "APIs" on the www. This sort of data should be on an FTP server. I can build my own "apps". Give me the option of raw data. Just my opinion, nothing more.

That's not quite what is going on. I had to do a paper for class recently that was looking at economic indicators. If I needed data on "Agriculture & Rural Development":

http://data.worldbank.org/topic/agriculture-and-rural-develo...

Down at the bottom is a link that will give you a csv file:

http://api.worldbank.org/v2/en/topic/1?downloadformat=csv

I thought it was really easy. I ended up having to visit multiple download links, but stitching the data together was simple using Python.

Without looking at the paper, I'm not sure what exactly they have. Have they just done the stitching already and are providing the complete data set? Or is there additional data not covered in the csv downloads?

Edit: Having looked a the paper it looks like the data they scraped was not in the CSVs. But I really cannot tell. If that is the case, I don't know why only some data is available as a bulk download and other data is not. So...back to your original point.

Re: We scraped the World Bank's website

#17

Post by keithpeter »

bybjorn wrote: post #14
dalek2point3 wrote: post #9

i thought the point of this way, gee, the world bank makes it hard for you to get data out, so we scraped them, and here is the data for you guys to play around with. I looked for a bit, but couldnt find the data -- am I missing something?

The last sentence on that page reads: "The full data can be downloaded at www.cgdev.org/povcalnet."

And the page linked has large number of files that appear to be relevant to other papers but not this one.

However, this link

http://www.cgdev.org/section/publications?f[0]=field_documen...

leads to page that has two papers listed by title. Click the second of the papers. Click Data and Analysis tab, and you can download the CSV files they obtained. You don't need a log in but you do need to agree to sensible looking conditions.

I can't seem to link directly to the agree terms/download page.

Re: We scraped the World Bank's website

#18

Post by unreal37 »

coldcode wrote: post #15

Why would such important public data be only available to internal researchers? Cost? Politics? Fear?

If 70% of their reports are downloaded only a couple of times, ever, I would guess that the answer is that nobody is interested in them. Have you ever read one?

Re: We scraped the World Bank's website

#19

Post by obstacle1 »

Looking at the source right now. Noticed comments in the code along the lines of "selenium is really slow traversing the dom" etc.. Also noticed the script implements the non-headless Firefox WebDriver. Wouldn't it have been much faster to have used GhostDriver or some similar headless solution?

Re: We scraped the World Bank's website

#20

Post by [deleted] »

aaron-lebo wrote: post #16
yp_all wrote: post #13

"public domain... but can [only] be accessed in small pieces" Sounds like the wonderful world of "APIs" on the www. This sort of data should be on an FTP server. I can build my own "apps". Give me the option of raw data. Just my opinion, nothing more.

That's not quite what is going on. I had to do a paper for class recently that was looking at economic indicators. If I needed data on "Agriculture & Rural Development": http://data.worldbank.org/topic/agriculture-and-rural-develo... Down at the bottom is a link that will give you a csv file: http://api.worldbank.org/v2/en/topic/1?downloadformat=csv I thought it was really easy. I ended up having to visit multiple do…

[deleted]
Post reply on HN