Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We scraped the World Bank's website

cgdev.org

Post reply on HN
Print view View original

21–30 of 37 posts

Re: We scraped the World Bank's website

#21

Post by alecco »

mileswu wrote: post #5

An interesting and somewhat related (but only tangentially) article I read a couple of days ago found that nearly 1/3rd of World Bank reports are never read, not even by a single person: http://www.washingtonpost.com/blogs/wonkblog/wp/2014/05/08/t... It was submitted to HN ( https://news.ycombinator.com/item?id=7715881 ) by another user, but probably never got traction because the title of the article is very vague.

Luckily there is TabulaPDF https://github.com/jazzido/tabula (Open source, made with a grant by the Knight Foundation)

A lot of praise from journalist in their twitter feed: https://twitter.com/TabulaPDF

Re: We scraped the World Bank's website

#23

Post by Zaheer »

The code is very well commented and I'd highly recommend if anyones interested in scraping to read through it. As a bonus, the Appendix details all the steps to run the script making it very easy for beginners.

Re: We scraped the World Bank's website

#24

Post by mburns »

yp_all wrote: post #13

"public domain... but can [only] be accessed in small pieces" Sounds like the wonderful world of "APIs" on the www. This sort of data should be on an FTP server. I can build my own "apps". Give me the option of raw data. Just my opinion, nothing more.

You're misquoting the article.

"...is not in the public domain, but can be accessed in small pieces..." (emphasis added)

Besides, this data should absolutely be provided in an API. An exporter tool that queries the API is superior to a static ftp site.

Re: We scraped the World Bank's website

#25

Post by [deleted] »

obstacle1 wrote: post #12
officialjunk wrote: post #7

Earlier quoted context omitted.

1. go here http://www.cgdev.org/section/publications?f[0]=field_documen... 2. click on the link to "We Just Ran Twenty-Three Million Queries of the World Bank's Web Site" 3. click on the "Data & Analysis" tab 4. scroll to the bottom and there are download links to harvester_parameters.py, harvester.py and unloader.py BUT i can't seem to actually download them, as there is some redirect that fails :( i tried creating…

Perfect, thanks. I got the redirect error too when I tried to download _all_ files. However when I deselected all and just checked the readme, csv data, and python scripts, it worked just fine.

[deleted]

Re: We scraped the World Bank's website

#26

Post by yp_all »

mburns wrote: post #24
yp_all wrote: post #13

"public domain... but can [only] be accessed in small pieces" Sounds like the wonderful world of "APIs" on the www. This sort of data should be on an FTP server. I can build my own "apps". Give me the option of raw data. Just my opinion, nothing more.

You're misquoting the article. "...is not in the public domain, but can be accessed in small pieces..." (emphasis added) Besides, this data should absolutely be provided in an API. An exporter tool that queries the API is superior to a static ftp site.

Sorry for the misquote. I guess that my misquote completely misrepresents the issue?

I never said the data should be not provided in an API.

If you read closely (for more than only "errors"), then you would observe the word "option".

That word is there for a reason.

The role of the FTP site (or whatever protocol you prefer) is to transfer the raw data in bulk to my local media.

Then I move it into my own database of choice and write my own code to access it.

Anyway, fear not. Your preferred www "API" world is not in jeopardy.

Cogent arguments why raw, bulk _public data_ _should never be provided_ in addition to rate-limited, by the slice snippets via "APIs" and third party "app developers" are welcome.

Re: We scraped the World Bank's website

#27

Post by tokenizerrr »

obstacle1 wrote: post #19

Looking at the source right now. Noticed comments in the code along the lines of "selenium is really slow traversing the dom" etc.. Also noticed the script implements the non-headless Firefox WebDriver. Wouldn't it have been much faster to have used GhostDriver or some similar headless solution?

Unless you need to evaluate JavaScript or take screenshots of the rendered page is there any point at all in using a webdriver like that instead of building a plain old scraper?

Re: We scraped the World Bank's website

#28

Post by [deleted] »

obstacle1 wrote: post #19

Looking at the source right now. Noticed comments in the code along the lines of "selenium is really slow traversing the dom" etc.. Also noticed the script implements the non-headless Firefox WebDriver. Wouldn't it have been much faster to have used GhostDriver or some similar headless solution?

[deleted]

Re: We scraped the World Bank's website

#29

Post by dmn001 »

tokenizerrr wrote: post #27
obstacle1 wrote: post #19

Looking at the source right now. Noticed comments in the code along the lines of "selenium is really slow traversing the dom" etc.. Also noticed the script implements the non-headless Firefox WebDriver. Wouldn't it have been much faster to have used GhostDriver or some similar headless solution?

Unless you need to evaluate JavaScript or take screenshots of the rendered page is there any point at all in using a webdriver like that instead of building a plain old scraper?

I agree, using Selenium seems like an unnecessary waste of time and cpu resources when replicating the GET/POST requests and parsing the html response using a simple Perl or Python script would have sufficed.
Post reply on HN