Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Web Scraping Is Revealing Lobbying and Corruption in Peru

blog.scrapinghub.com

Post reply on HN
Print view View original

41–50 of 80 posts

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#41

Post by carlosp420 »

nsoldiac wrote: post #37
carlosp420 wrote: post #9

Hi there, I am the author of the blog post. I will be happy to answer any question.

It would be interesting to see the volume of visits by government office year over year. I have a feeling that periods around elections might look very different. Also would be interested to see distribution color-coded by industry. Mining and contracting should pop up for certain time periods and government agencies.

yes. So far we have a very simple API http://manolo.rocks/docs/ With this API, it is possible to download all the structured data kept in Manolo and do such interesting analyses.

Or maybe that can be implemented in Manolo's GUI. It should not be difficult as it is based on Django.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#45

Post by [deleted] »

jorgecurio wrote: post #34
cookiecaper wrote: post #23

Earlier quoted context omitted.

Every time I see a scrapinghub post I ask the same question: what's your strategy for dealing with CFAA suits that arise from use of your platform? Most web scraping is illegal in the United States. I completely accept how important scraping is as a data source, but that doesn't make it any more legal. It's in a space right now where only big companies can take unmitigated advantage of the tool, because it'd cost mil…

> Most web scraping is illegal in the United States. Got a source on that? Google scrapes all the time. This is how they index all the pages it discovers. The only real scenario I recall is 3tap vs craigslist but they just kept scraping craigslist even after they banned their IP addresses with multiple proxies. Then there are airplane ticket websites scraping each other and getting into hot waters. Having said that i…

[deleted]

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#47

Post by sergiotapia »

carlosp420 wrote: post #9

Hi there, I am the author of the blog post. I will be happy to answer any question.

Existe alguna fuente de informacion como la de Peru, para Bolivia? Me imagino que hay mucho que descrubir sobre la corrupcion en Bolivia y el trafico de influencias.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#48

Post by rjeli »

carlosp420 wrote: post #40
nsoldiac wrote: post #30

Earlier quoted context omitted.

Carlos, super buen trabajo, felicitaciones!! Llevo tiempo estudiando temas relacionados a tecnología vs. corrupción desde acá en Berkeley. Tengo testimonios interesantes de contactos que han vivido el cambio post-tecnología en el gobierno. Perú tiene harto potencial en esta área. Si necesitas ayuda en cualquier momento feliz de apoyarte!

muchas gracias! En el Perú ya hay varios grupos de periodistas que se han asociado con programadores para hacer proyectos interesantes de periodismo de datos. Está Ojo Publico, Convoca e IDL reporteros. Pero igual no nos damos abasto hay tanto por hacer!

nsoldiac:

Carlos, super good job, congratulations!! I study issues related to technology vs. corruption from here in Berkeley. I have interesting evidence from contacts who have experienced post-technology change in government. Peru has (too much) potential in this area. If you need help at any time I am happy to support you!

carlosp420:

thank you! In Peru there are already several groups of journalists who have partnered with developers to make interesting data journalism projects. There is Ojo Publico [1], Convoca [2] and IDL Reporters [3]. But all the same, it's not enough, there's so much to do!

[1] http://ojo-publico.com/ [2] http://www.convoca.pe/ [3] https://idl-reporteros.pe/

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#49

Post by smarx007 »

alecco wrote: post #42

In other countries, corrupt politicians found out a simple captcha per n items is good enough to defeat analysis.

https://anti-captcha.com/ & https://rucaptcha.com/ - I think that can be best summarised as "from Russia with love" :)
Post reply on HN