Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Web Scraping Is Revealing Lobbying and Corruption in Peru

blog.scrapinghub.com

Post reply on HN
Print view View original

11–20 of 80 posts

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#11

Post by hackuser »

danso wrote: post #4

FWIW, if you live in the U.S., then you benefit from having such data in great quantity, though I don't think it's sliced-and-diced to near the potential that it has: Lobbyists have to follow registration procedures, and their official interactions and contributions are posted to an official database that can be downloaded as bulk XML: http://www.senate.gov/legislative/lobbyingdisc.htm#lobbyingd... Could they lie? Su…

Thanks; it's invaluable to hear from someone who has experience with the data.

Do you know what they are required to report? For example, if they have a 'social' dinner with a lobbyist, must that be reported? Are the requirements the same across the Executive Branch? All three branches?

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#12

Post by unsettledtck »

jorgecurio wrote: post #3

Really interesting use of data extraction.... For developers and managers out there, do you prefer to build your own in-house scrapers or use Scrapy or tools like Mozenda instead? What about import.io and kimono? I'm asking because lot of developers seem to be adamant against using web scraping tools they didn't develop themselves. Which seems counter productive because you are going into technical debt for an alread…

Full disclosure, I work for Scrapinghub. Our tools are Scrapy and Portia, both open source and both free as in beer. Scrapy is for those who want fine-tuned manual control and who have a background in Python. Portia is the visual web scraper for those who are non-technical to technical but don't want to bother with code.

Web scraping is everywhere, even if it's not necessarily spoken openly about or acknowledged. The publicized perception of web scraping is fairly negative, but doesn't take into account the benefits of data used in machine-learning or democratized data extraction (as in the case of this article or for building public service apps like transportation notifications), or the simple realities of competitive pricing and monitoring the activities of resellers.

Researchers, academics, data scientists, marketers, the list goes on for those who use web scraping daily.

Glad you enjoyed the article! I'm hoping that more examples of ethical data extraction will start to turn the tide of public perception.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#13

Post by gearhart »

carlosp420 wrote: post #9

Hi there, I am the author of the blog post. I will be happy to answer any question.

This is great work. Forgive me if I'm missing it, but since the blog post implies you're aggregating and cleaning the data from several lists, is there any way to see the latest additions (RSS etc?) rather than directly searching for individuals?

It would make it more useful for flagging up potential stories, as well as researching stories journalists are already writing.

disclosure: I work for a company that provides real-time data to journalists for story discovery, and I know we'd certainly be interested

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#14

Post by emluque »

jorgecurio wrote: post #3

Really interesting use of data extraction.... For developers and managers out there, do you prefer to build your own in-house scrapers or use Scrapy or tools like Mozenda instead? What about import.io and kimono? I'm asking because lot of developers seem to be adamant against using web scraping tools they didn't develop themselves. Which seems counter productive because you are going into technical debt for an alread…

I recently did a website, that mines Argentinian Central Bank statistics daily and generates graphics and reports: http://estadisticasbcra.com/en

( The data that I'm mining is published here: http://www.bcra.gov.ar/Estadisticas/estprv010000.asp )

On this case, some scripts using Beautiful Soup were enough to get the job done, but I was completely unaware of Scrapy, seems like a fantastic tool, if I would have known about it I probably would have used it.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#15

Post by unsettledtck »

jorgecurio wrote: post #7
austinhutch wrote: post #5

Earlier quoted context omitted.

After Kimono got shut down, I think a self-hosted open source version would be extremely popular. I want to build my own solution, but the API functionality and pagination / AJAX loaded data would be too difficult.

interesting, how would a self-hosted open source version make money tho in order to support itself and continue to upgrade? Is this even a realistic business model? Seems like this is what Scrapy is doing and what Import.io is doing. Make the tool free in order to get free marketing and then charge people willing to pay money to extract data. Meanwhile I see Mozenda charging like 5 cents for each page extracted, do y…

So for Scrapy and Portia, they are both free as in beer, specifically because we believe in the power of open source. Scrapy actually precedes Scrapinghub and was certainly not developed as a marketing tool.

Charges come with large scale crawls (above certain limits on our platform), additional products like Crawlera (our smart downloader that routes requests from a crawl through a pool of IP addresses to avoid bans), datasets, and for us to handle complex crawls for companies outsourcing to us.

Our model is that there is something for everyone whether you are looking to dip your toes into web scraping (free), use it occasionally (usually journalists) or dependent on web crawling for your business.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#16

Post by carlosp420 »

gearhart wrote: post #13
carlosp420 wrote: post #9

Hi there, I am the author of the blog post. I will be happy to answer any question.

This is great work. Forgive me if I'm missing it, but since the blog post implies you're aggregating and cleaning the data from several lists, is there any way to see the latest additions (RSS etc?) rather than directly searching for individuals? It would make it more useful for flagging up potential stories, as well as researching stories journalists are already writing. disclosure: I work for a company that provide…

I never thought of that, but certainly having a RSS feed is a great idea. I have not done it as the journalists have not requested it. So far they have been asking me for more spiders so Manolo would include visit records from other Peruvian institutions.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#17

Post by unsettledtck »

austinhutch wrote: post #5
jorgecurio wrote: post #3

Really interesting use of data extraction.... For developers and managers out there, do you prefer to build your own in-house scrapers or use Scrapy or tools like Mozenda instead? What about import.io and kimono? I'm asking because lot of developers seem to be adamant against using web scraping tools they didn't develop themselves. Which seems counter productive because you are going into technical debt for an alread…

After Kimono got shut down, I think a self-hosted open source version would be extremely popular. I want to build my own solution, but the API functionality and pagination / AJAX loaded data would be too difficult.

Not sure if you're interested, but we (Scrapinghub) do offer a Kimono to Portia migration https://blog.scrapinghub.com/2016/02/25/migrate-your-kimono-...

Otherwise, I'd recommend you check out Portia (open source). We're in the middle of releasing the beta 2.0 version.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#18

Post by benologist »

ecthiender wrote: post #2

Very interesting, how tools like these can be so much helpful for journalists and generally transparency in government functions. Probably world changing, when considering that even semi-technical folks can cook up tools to dig into things like this. I know this tool was by a developer, but scrapinghub has web UI to make scrapers.

A similar thing happened in Costa Rica -

    “You can’t visit 160,000 people,” she notes. “But 
    you can easily interrogate 160,000 records.”
http://foreignpolicy.com/2015/05/27/the-data-sleuths-of-san-...

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#19

Post by malka »

jorgecurio wrote: post #7
austinhutch wrote: post #5

Earlier quoted context omitted.

After Kimono got shut down, I think a self-hosted open source version would be extremely popular. I want to build my own solution, but the API functionality and pagination / AJAX loaded data would be too difficult.

interesting, how would a self-hosted open source version make money tho in order to support itself and continue to upgrade? Is this even a realistic business model? Seems like this is what Scrapy is doing and what Import.io is doing. Make the tool free in order to get free marketing and then charge people willing to pay money to extract data. Meanwhile I see Mozenda charging like 5 cents for each page extracted, do y…

by proposing paid hosting and support for companies that don't want the burden to manage it themselves ? There could be some additional features with the paid version also

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#20

Post by unsettledtck »

danso wrote: post #4

FWIW, if you live in the U.S., then you benefit from having such data in great quantity, though I don't think it's sliced-and-diced to near the potential that it has: Lobbyists have to follow registration procedures, and their official interactions and contributions are posted to an official database that can be downloaded as bulk XML: http://www.senate.gov/legislative/lobbyingdisc.htm#lobbyingd... Could they lie? Su…

I just ran across https://www.opensecrets.org/ and found it quite useful and comprehensive in tracking contributions to candidates.

I live in the US and am privileged with the level of transparency that exists, but it's still not necessarily enough. Similar issues are present with the clunky nature of government websites and databases and so I think we're in agreement that it's not even close to the potential of what it could be.

Thanks for sharing all the links and information!

Post reply on HN