Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Web Scraping Is Revealing Lobbying and Corruption in Peru

blog.scrapinghub.com

Post reply on HN
Print view View original

31–40 of 80 posts

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#31

Post by predius »

jorgecurio wrote: post #3

Really interesting use of data extraction.... For developers and managers out there, do you prefer to build your own in-house scrapers or use Scrapy or tools like Mozenda instead? What about import.io and kimono? I'm asking because lot of developers seem to be adamant against using web scraping tools they didn't develop themselves. Which seems counter productive because you are going into technical debt for an alread…

I agree, this is definitely a solved problem!

If you need to build a solid web scraping stack which is going to be maintained by many people and is critical to your business, you have two options… to use Scrapy or to build something yourself.

Scrapy has been tried and tested over 6-7 years of community development, as well as being the base infrastructure for a number of >$1B businesses. Not only this, but there is a suite of tools which you have been built around it – Portia for one, but also other lots of useful open source libraries: http://scrapinghub.com/opensource/).

Right now most people still have the issue of having to use xpath or css selectors to run your crawl or get the data, but not for too long.

There's more and more ways of skipping this step and getting at data automatically: https://github.com/redapple/parslepy/wiki/Use-parslepy-with-... https://speakerdeck.com/amontalenti/web-crawling-and-metadat... https://github.com/scrapy/loginform https://github.com/TeamHG-Memex/Formasaurus https://github.com/scrapy/scrapely https://github.com/scrapinghub/webpager https://moz.com/devblog/benchmarking-python-content-extracti...

Scrapy (and also lots of python tools, likely a majority of them created by people using it and BeautifulSoup) have lowered the cost of building web data harvesting systems to the point where one guy can build crawlers for an entire industry in a couple of months.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#32

Post by RodericDay »

nsoldiac wrote: post #30
carlosp420 wrote: post #9

Hi there, I am the author of the blog post. I will be happy to answer any question.

Carlos, super buen trabajo, felicitaciones!! Llevo tiempo estudiando temas relacionados a tecnología vs. corrupción desde acá en Berkeley. Tengo testimonios interesantes de contactos que han vivido el cambio post-tecnología en el gobierno. Perú tiene harto potencial en esta área. Si necesitas ayuda en cualquier momento feliz de apoyarte!

Igualmente. También vivo en el extranjero (Canada), pero estoy totalmente dispuesto a apoyar en cualquier iniciativa de este tipo.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#34

Post by jorgecurio »

cookiecaper wrote: post #23
unsettledtck wrote: post #12

Earlier quoted context omitted.

Full disclosure, I work for Scrapinghub. Our tools are Scrapy and Portia, both open source and both free as in beer. Scrapy is for those who want fine-tuned manual control and who have a background in Python. Portia is the visual web scraper for those who are non-technical to technical but don't want to bother with code. Web scraping is everywhere, even if it's not necessarily spoken openly about or acknowledged. The…

Every time I see a scrapinghub post I ask the same question: what's your strategy for dealing with CFAA suits that arise from use of your platform? Most web scraping is illegal in the United States. I completely accept how important scraping is as a data source, but that doesn't make it any more legal. It's in a space right now where only big companies can take unmitigated advantage of the tool, because it'd cost mil…

> Most web scraping is illegal in the United States.

Got a source on that? Google scrapes all the time. This is how they index all the pages it discovers.

The only real scenario I recall is 3tap vs craigslist but they just kept scraping craigslist even after they banned their IP addresses with multiple proxies.

Then there are airplane ticket websites scraping each other and getting into hot waters.

Having said that it's not a clear cut definition as you'd like to put it. CFAA ruling was only because craigslist felt directly threatened by Padmapper which relied on 3taps.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#35

Post by jorgecurio »

malka wrote: post #19
jorgecurio wrote: post #7

Earlier quoted context omitted.

interesting, how would a self-hosted open source version make money tho in order to support itself and continue to upgrade? Is this even a realistic business model? Seems like this is what Scrapy is doing and what Import.io is doing. Make the tool free in order to get free marketing and then charge people willing to pay money to extract data. Meanwhile I see Mozenda charging like 5 cents for each page extracted, do y…

by proposing paid hosting and support for companies that don't want the burden to manage it themselves ? There could be some additional features with the paid version also

I'm wondering about this but how realistic do you expect someone to pay when it's already free? I would imagine only large enterprise users so essentially you are supporting free users by charging enterprise users that may pay you for support.

Horrible industry imho when you have to give away things for free just to be competitive. I just don't get why people would expect software to be free.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#36

Post by dang »

NiftyFifty wrote: post #33

Pussies.

We've banned this account for repeatedly violating the HN guidelines.

We're happy to unban accounts when people give us reason to believe they will post only civil and substantive comments in the future. You're welcome to email hn@ycombinator.com if that's the case.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#37

Post by nsoldiac »

carlosp420 wrote: post #9

Hi there, I am the author of the blog post. I will be happy to answer any question.

It would be interesting to see the volume of visits by government office year over year. I have a feeling that periods around elections might look very different. Also would be interested to see distribution color-coded by industry. Mining and contracting should pop up for certain time periods and government agencies.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#38

Post by predius »

jorgecurio wrote: post #35
malka wrote: post #19

Earlier quoted context omitted.

by proposing paid hosting and support for companies that don't want the burden to manage it themselves ? There could be some additional features with the paid version also

I'm wondering about this but how realistic do you expect someone to pay when it's already free? I would imagine only large enterprise users so essentially you are supporting free users by charging enterprise users that may pay you for support. Horrible industry imho when you have to give away things for free just to be competitive. I just don't get why people would expect software to be free.

We don't expect anyone to pay for Scrapy or Portia!

We provide the best Platform (as a Service) to run Scrapy or Portia spiders, and will soon be supporting most standard web scraping technologies. This is free for light users, but we charge for people who need extra or dedicated computing or network resources.

We also provide help to startups or enterprise orgs looking to get help in building a web data harvesting system (more than just parsing pages!), either by building it ourselves or by helping our partners train their engineers in using our technologies.

This has worked so far, and we're very healthy from a revenue perspective – more than doubling every year for a few years now, and good enough to grow to become the largest fully distributed company outside of the US.

We're pretty happy with being a brand that gives to the community, it tends to get repaid 10x in the long run.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#39

Post by [deleted] »

RodericDay wrote: post #32
nsoldiac wrote: post #30

Earlier quoted context omitted.

Carlos, super buen trabajo, felicitaciones!! Llevo tiempo estudiando temas relacionados a tecnología vs. corrupción desde acá en Berkeley. Tengo testimonios interesantes de contactos que han vivido el cambio post-tecnología en el gobierno. Perú tiene harto potencial en esta área. Si necesitas ayuda en cualquier momento feliz de apoyarte!

Igualmente. También vivo en el extranjero (Canada), pero estoy totalmente dispuesto a apoyar en cualquier iniciativa de este tipo.

[deleted]

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#40

Post by carlosp420 »

nsoldiac wrote: post #30
carlosp420 wrote: post #9

Hi there, I am the author of the blog post. I will be happy to answer any question.

Carlos, super buen trabajo, felicitaciones!! Llevo tiempo estudiando temas relacionados a tecnología vs. corrupción desde acá en Berkeley. Tengo testimonios interesantes de contactos que han vivido el cambio post-tecnología en el gobierno. Perú tiene harto potencial en esta área. Si necesitas ayuda en cualquier momento feliz de apoyarte!

muchas gracias! En el Perú ya hay varios grupos de periodistas que se han asociado con programadores para hacer proyectos interesantes de periodismo de datos. Está Ojo Publico, Convoca e IDL reporteros. Pero igual no nos damos abasto hay tanto por hacer!
Post reply on HN