We scraped the World Bank's website • Hacker News Board

Re: We scraped the World Bank's website

#31

Post by frozenport » Sun, May 11, 2014, 2:12 AM UTC

ohashi wrote: post #11

mileswu wrote: post #5

An interesting and somewhat related (but only tangentially) article I read a couple of days ago found that nearly 1/3rd of World Bank reports are never read, not even by a single person: http://www.washingtonpost.com/blogs/wonkblog/wp/2014/05/08/t... It was submitted to HN ( https://news.ycombinator.com/item?id=7715881 ) by another user, but probably never got traction because the title of the article is very vague.

It sounds like a marketing problem. Although, I suspect the PDFs aren't the only dissemination method (as stated in the article). After someone spends so much time writing a report, it probably sits in their consciousness and spreads via the person's other interactions (eg shaping their perspective, thinking, and pursuits). With so many people producing so much content on a daily basis, it's hard to imagine people ac…

Or maybe they are spreading bad ideas, we won't know unless somebody reviews them.

Re: We scraped the World Bank's website

#32

Post by obstacle1 » Sun, May 11, 2014, 2:57 AM UTC

tokenizerrr wrote: post #27

obstacle1 wrote: post #19

Looking at the source right now. Noticed comments in the code along the lines of "selenium is really slow traversing the dom" etc.. Also noticed the script implements the non-headless Firefox WebDriver. Wouldn't it have been much faster to have used GhostDriver or some similar headless solution?

Unless you need to evaluate JavaScript or take screenshots of the rendered page is there any point at all in using a webdriver like that instead of building a plain old scraper?

Great point.

Re: We scraped the World Bank's website

#33

Post by nickthemagicman » Sun, May 11, 2014, 3:57 AM UTC

I'm not even sure what kind of data this would be but it sounds fascinating.

Re: We scraped the World Bank's website

#34

Post by e12e » Sun, May 11, 2014, 10:43 AM UTC

Looking at some random datasets, this looks like it's pretty small datasets (less than a mb?) -- so it'd be nice to just have them zip-ed and available as a torrent (say plain-text description in one file, and csv in a folder per set)?

Rather than re-creating the problem of data being hard to find by splitting the download links over two-pages, and apparently requiring a click-through for every dataset to get at the data?

Re: We scraped the World Bank's website

#35

Post by iand » Sun, May 11, 2014, 10:57 PM UTC

I guess they could have just gone to http://povertydata.worldbank.org/poverty/home/ and downloaded via CSV

Re: We scraped the World Bank's website

#36

Post by [deleted] » Mon, May 12, 2014, 12:33 AM UTC

yp_all wrote: post #26

mburns wrote: post #24

Earlier quoted context omitted.

You're misquoting the article. "...is not in the public domain, but can be accessed in small pieces..." (emphasis added) Besides, this data should absolutely be provided in an API. An exporter tool that queries the API is superior to a static ftp site.

Sorry for the misquote. I guess that my misquote completely misrepresents the issue? I never said the data should be not provided in an API. If you read closely (for more than only "errors"), then you would observe the word "option". That word is there for a reason. The role of the FTP site (or whatever protocol you prefer) is to transfer the raw data in bulk to my local media. Then I move it into my own database of…

[deleted]

Re: We scraped the World Bank's website

#37

Post by bendykstra » Mon, May 12, 2014, 12:44 PM UTC

obstacle1 wrote: post #19

Looking at the source right now. Noticed comments in the code along the lines of "selenium is really slow traversing the dom" etc.. Also noticed the script implements the non-headless Firefox WebDriver. Wouldn't it have been much faster to have used GhostDriver or some similar headless solution?

Hi, one of the authors here. First, though I worked on the paper, I am not employed by CGD, so these comments are my own.

The main reason for using the non-headless Firefox WebDriver was that we wanted the script to access the site just like a human user. This made it easy to explain to non-technical people exactly how we had gotten the data. We didn't want to do anything that could be seen as circumventing the interface that the World Bank had created for that purpose.

Up to a point, performance was not a concern. In fact, as slow as Selenium is, we still artificially limited the speed of the script by waiting three seconds between each set of queries. However, when it came to selecting options, it could take Selenium tens of seconds, so that was done with js.