Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Web Scraping Is Revealing Lobbying and Corruption in Peru

blog.scrapinghub.com

Post reply on HN
Print view View original

21–30 of 80 posts

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#21

Post by danso »

hackuser wrote: post #11
danso wrote: post #4

FWIW, if you live in the U.S., then you benefit from having such data in great quantity, though I don't think it's sliced-and-diced to near the potential that it has: Lobbyists have to follow registration procedures, and their official interactions and contributions are posted to an official database that can be downloaded as bulk XML: http://www.senate.gov/legislative/lobbyingdisc.htm#lobbyingd... Could they lie? Su…

Thanks; it's invaluable to hear from someone who has experience with the data. Do you know what they are required to report? For example, if they have a 'social' dinner with a lobbyist, must that be reported? Are the requirements the same across the Executive Branch? All three branches?

I don't have much experience with the lobbying rules except for times that I've had to research things specifically. Usually disclosure requirements come with a minimum amount...In the House (not sure if the exact limits apply to the Senate...), the ethics rules are quite strict but not everything is recorded...for example, a legislator (or their staff) can only receive $100 of gifts from a single source in a calendar year..."gifts" being basically anything of value...but things under $10 don't count toward that limit. So getting Frappuccinos everyday with your favorite CEO probably wouldn't be recorded in any official capacity even though not only do those add up monetarily, but someone getting coffee with a legislator on a frequent basis would be a huge point of potential influence. However, legislators aren't allowed to get gifts (such as paid dinners) at all from a registered lobbyist [1].

Both the House and the Senate have gift travel databases (travel that's reimbursed by an outside group, such as a charter flight to visit an oil drilling rig) [2]

The branches differ in how such things are reported...this was pretty obvious recently when Justice Scalia died at a ranch and people started wondering who paid for the trip...take one look at how these forms are supplied and it should be pretty obvious why we don't normally hear about SCOTUS relationships until something really weird happens [3].

This NYT editorial "So Who's a Lobbyist?" has a nice rundown of the ways that people who would generally be considered a lobbyist can escape disclosure requirements: http://www.nytimes.com/2012/01/27/opinion/so-whos-a-lobbyist...

Still, it's useful to be able to parse the dataset in an attempt to find what's missing...something that is difficult to do conceptually unless you're dealing with the actual dataset on your own system.

[1] https://ethics.house.gov/gifts/house-gift-rule

[2] http://clerk.house.gov/public_disc/giftTravel.aspx

[3] http://pfds.opensecrets.org/N99999918_2008.pdf

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#22

Post by gearhart »

carlosp420 wrote: post #16
gearhart wrote: post #13

Earlier quoted context omitted.

This is great work. Forgive me if I'm missing it, but since the blog post implies you're aggregating and cleaning the data from several lists, is there any way to see the latest additions (RSS etc?) rather than directly searching for individuals? It would make it more useful for flagging up potential stories, as well as researching stories journalists are already writing. disclosure: I work for a company that provide…

I never thought of that, but certainly having a RSS feed is a great idea. I have not done it as the journalists have not requested it. So far they have been asking me for more spiders so Manolo would include visit records from other Peruvian institutions.

If you do choose to implement it, please do let me know (email in profile), I'll definitely make sure we use it.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#23

Post by cookiecaper »

unsettledtck wrote: post #12
jorgecurio wrote: post #3

Really interesting use of data extraction.... For developers and managers out there, do you prefer to build your own in-house scrapers or use Scrapy or tools like Mozenda instead? What about import.io and kimono? I'm asking because lot of developers seem to be adamant against using web scraping tools they didn't develop themselves. Which seems counter productive because you are going into technical debt for an alread…

Full disclosure, I work for Scrapinghub. Our tools are Scrapy and Portia, both open source and both free as in beer. Scrapy is for those who want fine-tuned manual control and who have a background in Python. Portia is the visual web scraper for those who are non-technical to technical but don't want to bother with code. Web scraping is everywhere, even if it's not necessarily spoken openly about or acknowledged. The…

Every time I see a scrapinghub post I ask the same question: what's your strategy for dealing with CFAA suits that arise from use of your platform? Most web scraping is illegal in the United States.

I completely accept how important scraping is as a data source, but that doesn't make it any more legal. It's in a space right now where only big companies can take unmitigated advantage of the tool, because it'd cost millions of dollars to successfully defend a CFAA suit.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#24

Post by logn »

jorgecurio wrote: post #3

Really interesting use of data extraction.... For developers and managers out there, do you prefer to build your own in-house scrapers or use Scrapy or tools like Mozenda instead? What about import.io and kimono? I'm asking because lot of developers seem to be adamant against using web scraping tools they didn't develop themselves. Which seems counter productive because you are going into technical debt for an alread…

> an already solved problem

It's a hard problem to generalize.

> balance between people who want to scrape Linkedin to spam people, others looking to do good with the data they scrape, and website owners who get aggressive and threatening when they realize they are getting scraped

Agreed. No one wants to be the bad guy and most clients looking to spam people are awful clients to have anyhow. Btw scraping LinkedIn is fairly difficult/expensive and they like to sue people.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#26

Post by justinlardinois »

danso wrote: post #4

FWIW, if you live in the U.S., then you benefit from having such data in great quantity, though I don't think it's sliced-and-diced to near the potential that it has: Lobbyists have to follow registration procedures, and their official interactions and contributions are posted to an official database that can be downloaded as bulk XML: http://www.senate.gov/legislative/lobbyingdisc.htm#lobbyingd... Could they lie? Su…

Damn. This is pretty impressive.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#27

Post by [deleted] »

cookiecaper wrote: post #23
unsettledtck wrote: post #12

Earlier quoted context omitted.

Full disclosure, I work for Scrapinghub. Our tools are Scrapy and Portia, both open source and both free as in beer. Scrapy is for those who want fine-tuned manual control and who have a background in Python. Portia is the visual web scraper for those who are non-technical to technical but don't want to bother with code. Web scraping is everywhere, even if it's not necessarily spoken openly about or acknowledged. The…

Every time I see a scrapinghub post I ask the same question: what's your strategy for dealing with CFAA suits that arise from use of your platform? Most web scraping is illegal in the United States. I completely accept how important scraping is as a data source, but that doesn't make it any more legal. It's in a space right now where only big companies can take unmitigated advantage of the tool, because it'd cost mil…

[deleted]

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#28

Post by predius »

cookiecaper wrote: post #23
unsettledtck wrote: post #12

Earlier quoted context omitted.

Full disclosure, I work for Scrapinghub. Our tools are Scrapy and Portia, both open source and both free as in beer. Scrapy is for those who want fine-tuned manual control and who have a background in Python. Portia is the visual web scraper for those who are non-technical to technical but don't want to bother with code. Web scraping is everywhere, even if it's not necessarily spoken openly about or acknowledged. The…

Every time I see a scrapinghub post I ask the same question: what's your strategy for dealing with CFAA suits that arise from use of your platform? Most web scraping is illegal in the United States. I completely accept how important scraping is as a data source, but that doesn't make it any more legal. It's in a space right now where only big companies can take unmitigated advantage of the tool, because it'd cost mil…

Hi! This is definitely not legal advice, so consult with a lawyer and do your own research if you are thinking of applying this to your own practices.

I work for Scrapinghub as well and try to understand the law around this. I can help with some pointers to why I think some web scraping isn't illegal… there are of courses some limits to this.

When the data scraped is "is publicly available on the Internet, without requiring any login, password, or other individualized grant of access", the Eastern District Court of Virginia in Cvent vs Eventbrite (https://casetext.com/case/cvent-inc-v-eventbrite) ruled one could not be deemed to be exceeding unauthorized access.

There are two ways, that I know of, that courts have ruled you can exceed your authorization:

- When the site owner has contacted you and removed your authorization in a written manner, as happened on Craigslist vs 3taps.

- By accepting the terms of service and agreeing against scraping. You have to do this through a "clickwrap" ToS, rather than a "browsewrap". You can read about the differences here: https://termsfeed.com/blog/browsewrap-clickwrap/

As a matter of policy, we don't scrape any site with a ToS with clear anti-scraping language and which forces us to create an account or "constructively agree" as part of the use of the site.

Any user wishing to revoke authorization for anyone using our platform can make an abuse report on our site– we tend to handle these within 24 hours and haven't had a single claim go further than this stage, as we aim to be reasonable and look for a way to provide value to both sides.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#29

Post by predius »

gearhart wrote: post #22
carlosp420 wrote: post #16

Earlier quoted context omitted.

I never thought of that, but certainly having a RSS feed is a great idea. I have not done it as the journalists have not requested it. So far they have been asking me for more spiders so Manolo would include visit records from other Peruvian institutions.

If you do choose to implement it, please do let me know (email in profile), I'll definitely make sure we use it.

Should be a fairly easy thing to do, this is Python and Scrapy!

http://stackoverflow.com/questions/28127396/creating-rss-wit...

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#30

Post by nsoldiac »

carlosp420 wrote: post #9

Hi there, I am the author of the blog post. I will be happy to answer any question.

Carlos, super buen trabajo, felicitaciones!! Llevo tiempo estudiando temas relacionados a tecnología vs. corrupción desde acá en Berkeley. Tengo testimonios interesantes de contactos que han vivido el cambio post-tecnología en el gobierno. Perú tiene harto potencial en esta área. Si necesitas ayuda en cualquier momento feliz de apoyarte!
Post reply on HN