How Web Scraping Is Revealing Lobbying and Corruption in Peru • Hacker News Board

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#61

Post by ericd » Thu, Mar 10, 2016, 5:49 PM UTC

jorgecurio wrote: post #34

cookiecaper wrote: post #23

Earlier quoted context omitted.

Every time I see a scrapinghub post I ask the same question: what's your strategy for dealing with CFAA suits that arise from use of your platform? Most web scraping is illegal in the United States. I completely accept how important scraping is as a data source, but that doesn't make it any more legal. It's in a space right now where only big companies can take unmitigated advantage of the tool, because it'd cost mil…

> Most web scraping is illegal in the United States. Got a source on that? Google scrapes all the time. This is how they index all the pages it discovers. The only real scenario I recall is 3tap vs craigslist but they just kept scraping craigslist even after they banned their IP addresses with multiple proxies. Then there are airplane ticket websites scraping each other and getting into hot waters. Having said that i…

They didn't claim CFAA on us (PadMapper), and there was definitely no ruling on it (all parties settled). Just for the record.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#62

Post by jorgecurio » Thu, Mar 10, 2016, 6:13 PM UTC

cookiecaper wrote: post #54

predius wrote: post #28

Earlier quoted context omitted.

Hi! This is definitely not legal advice, so consult with a lawyer and do your own research if you are thinking of applying this to your own practices. I work for Scrapinghub as well and try to understand the law around this. I can help with some pointers to why I think some web scraping isn't illegal… there are of courses some limits to this. When the data scraped is "is publicly available on the Internet, without re…

Thanks for the response. It is good to know that you're cognizant of these issues, at least. Most websites have anti-scraping boilerplate in their ToU. I'm pretty sure that's in the "customized" ToU I got from LegalZoom. So you're basically saying that if the data is not behind a login and doesn't require you to fill any forms that contain either a checkbox or nearby language that indicates submitting the form consti…

everyone of those cases involve multi-people organizations with significant revenues or funding. These cases are largely reflecting of businesses forcefully shutting down other innovators by claiming some bullshit like CFAA. CFAA should really only apply to people who are doing SQL injections and other penetration. Vast majority of people scraping data do not fall under this category of malevolent behavior, although the dumbassery of people who wants to scrape linkedin for 30 bucks on freelancer ruin it for everyone.

I'm not sure Scrapinghub is funded externally as I couldn't find anything on their valuation or revenues so I assume that they are bootsrapped.

I do not discount any of what you wrote but a lot of it are imagined dangers, scrapinghub would be immune to such cases unless they sided with their customers like 3taps did. 3taps did not stop scraping for their client Padmapper. I don't think scrapinghub is willing to put their neck out for someone paying $20/month to scrape craigslist. In fact, those are the shittiest segments of this market imo, the bottom feeders who demand excessively unrealistic expectations from scraping as a blackbox magical world that will solve their problems.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#63

Post by cookiecaper » Thu, Mar 10, 2016, 6:13 PM UTC

ericd wrote: post #61

jorgecurio wrote: post #34

Earlier quoted context omitted.

> Most web scraping is illegal in the United States. Got a source on that? Google scrapes all the time. This is how they index all the pages it discovers. The only real scenario I recall is 3tap vs craigslist but they just kept scraping craigslist even after they banned their IP addresses with multiple proxies. Then there are airplane ticket websites scraping each other and getting into hot waters. Having said that i…

They didn't claim CFAA on us (PadMapper), and there was definitely no ruling on it (all parties settled). Just for the record.

Are you stating that there was no CFAA claim, or that PadMapper wasn't the involved party, because it was actually 3Taps? The case against 3Taps definitely included a CFAA claim and the judge refused to dismiss it.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#64

Post by jorgecurio » Thu, Mar 10, 2016, 6:15 PM UTC

ericd wrote: post #61

jorgecurio wrote: post #34

Earlier quoted context omitted.

> Most web scraping is illegal in the United States. Got a source on that? Google scrapes all the time. This is how they index all the pages it discovers. The only real scenario I recall is 3tap vs craigslist but they just kept scraping craigslist even after they banned their IP addresses with multiple proxies. Then there are airplane ticket websites scraping each other and getting into hot waters. Having said that i…

They didn't claim CFAA on us (PadMapper), and there was definitely no ruling on it (all parties settled). Just for the record.

WOW. so the guys doing all the heavy lifting (3taps) took all the heat in the end. So looks like 3taps is out of business but padmapper is still up and running....getting data from crowdsourcing? It's really odd that if you made this efficient by automating it then it's hacking.

This really is a shitty shitty business model. All that work 3taps did for you guys and they take all the heat? I don't know why 3taps didn't just comply, was PadMapper 100% of their business?

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#65

Post by jorgecurio » Thu, Mar 10, 2016, 6:19 PM UTC

cookiecaper wrote: post #60

jorgecurio wrote: post #34

Earlier quoted context omitted.

> Most web scraping is illegal in the United States. Got a source on that? Google scrapes all the time. This is how they index all the pages it discovers. The only real scenario I recall is 3tap vs craigslist but they just kept scraping craigslist even after they banned their IP addresses with multiple proxies. Then there are airplane ticket websites scraping each other and getting into hot waters. Having said that i…

>Got a source on that? The source is the CFAA, which makes it a crime and/or a tort to commit any "unauthorized" access to a computer system. Because authorization is not defined in the statute, it's a matter of interpretation whether or not one's use is unauthorized. Historically, judges have strongly disfavored scrapers. Most boilerplate Terms of Use contain language that forbids all "spiders, scrapers, bots, and a…

The tricky thing is that a tool or service provider of scraping if compliant to the demands of website owners to stop scraping, there is very little to claim damages. Even if the customer used scrapinghub to login to websites and scrape all the emails, all scrapinghub would need to do is hand over their customer on a silver platter. This is what the DMCA is for. Can you imagine if you manufactured a bicycle and somebody used it to commit a crime? Plausible deniability. Scrapinghub can't monitor everyone's usage all the time to make sure they are following each websites TOS (which are not legally binding).

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#66

Post by jorgecurio » Thu, Mar 10, 2016, 6:19 PM UTC

predius wrote: post #38

jorgecurio wrote: post #35

Earlier quoted context omitted.

I'm wondering about this but how realistic do you expect someone to pay when it's already free? I would imagine only large enterprise users so essentially you are supporting free users by charging enterprise users that may pay you for support. Horrible industry imho when you have to give away things for free just to be competitive. I just don't get why people would expect software to be free.

We don't expect anyone to pay for Scrapy or Portia! We provide the best Platform (as a Service) to run Scrapy or Portia spiders, and will soon be supporting most standard web scraping technologies. This is free for light users, but we charge for people who need extra or dedicated computing or network resources. We also provide help to startups or enterprise orgs looking to get help in building a web data harvesting s…

how much are you generating in terms of revenue? are you venture funded?

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#67

Post by ericd » Thu, Mar 10, 2016, 6:29 PM UTC

cookiecaper wrote: post #63

ericd wrote: post #61

Earlier quoted context omitted.

They didn't claim CFAA on us (PadMapper), and there was definitely no ruling on it (all parties settled). Just for the record.

Are you stating that there was no CFAA claim, or that PadMapper wasn't the involved party, because it was actually 3Taps? The case against 3Taps definitely included a CFAA claim and the judge refused to dismiss it.

You're right that there was a CFAA claim, but there wasn't one made against us.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#68

Post by ericd » Thu, Mar 10, 2016, 6:35 PM UTC

jorgecurio wrote: post #64

ericd wrote: post #61

Earlier quoted context omitted.

They didn't claim CFAA on us (PadMapper), and there was definitely no ruling on it (all parties settled). Just for the record.

WOW. so the guys doing all the heavy lifting (3taps) took all the heat in the end. So looks like 3taps is out of business but padmapper is still up and running....getting data from crowdsourcing? It's really odd that if you made this efficient by automating it then it's hacking. This really is a shitty shitty business model. All that work 3taps did for you guys and they take all the heat? I don't know why 3taps didn'…

No worries, just trying to add some nuance. Probably can't share much there.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#69

Post by cookiecaper » Thu, Mar 10, 2016, 6:41 PM UTC

jorgecurio wrote: post #65

cookiecaper wrote: post #60

Earlier quoted context omitted.

>Got a source on that? The source is the CFAA, which makes it a crime and/or a tort to commit any "unauthorized" access to a computer system. Because authorization is not defined in the statute, it's a matter of interpretation whether or not one's use is unauthorized. Historically, judges have strongly disfavored scrapers. Most boilerplate Terms of Use contain language that forbids all "spiders, scrapers, bots, and a…

The tricky thing is that a tool or service provider of scraping if compliant to the demands of website owners to stop scraping, there is very little to claim damages. Even if the customer used scrapinghub to login to websites and scrape all the emails, all scrapinghub would need to do is hand over their customer on a silver platter. This is what the DMCA is for. Can you imagine if you manufactured a bicycle and someb…

The DMCA protects service providers from copyright claims for user-generated content as long as they comply with takedown requests, etc. Scrapinghub may have a defense to copyright claims there (though I seriously doubt it due to the nature of their relationship with the customer; they're not a DMCA "safe harbor" and the data they're using isn't user-generated content), but not to CFAA claims.

It's illegal to break the CFAA whether the plaintiff specifically tells you that they think you're doing it or not. If they send a C&D, yes, you'd be wise to comply, but that's not going to absolve you from claims that you harmed their company by violating the CFAA before they sent it (which do happen and are usually claiming a pretty ridiculously silly amount of damages for something as innocent as downloading a web page from their server). You'd have to argue in court that your access was authorized and they'd have to argue that your access wasn't authorized. The judge and/or jury would then evaluate.

3Taps was actually quite similar to Scrapinghub. I don't think they have as much of a defense as you'd like. And Terms of Use are actually usually considered legally binding; to the extent that they're not, it's usually because of something minor like not putting the notice that you agree to the ToU by using the site in plain view.

Re: How Web Scraping Is Revealing Lobbying and Corruption in Peru

#70

Post by vram22 » Thu, Mar 10, 2016, 6:56 PM UTC

unsettledtck wrote: post #15

jorgecurio wrote: post #7

Earlier quoted context omitted.

interesting, how would a self-hosted open source version make money tho in order to support itself and continue to upgrade? Is this even a realistic business model? Seems like this is what Scrapy is doing and what Import.io is doing. Make the tool free in order to get free marketing and then charge people willing to pay money to extract data. Meanwhile I see Mozenda charging like 5 cents for each page extracted, do y…

So for Scrapy and Portia, they are both free as in beer, specifically because we believe in the power of open source. Scrapy actually precedes Scrapinghub and was certainly not developed as a marketing tool. Charges come with large scale crawls (above certain limits on our platform), additional products like Crawlera (our smart downloader that routes requests from a crawl through a pool of IP addresses to avoid bans)…

>Scrapy actually precedes Scrapinghub

Right. I had first come across Scrapy, while browsing the web for Python software tools, some years ago, on the site of a company in Uruguay called Insophia. It was in the list of products developed by them, and that they worked on. Scrapinghub came later.