Ask HN: Who uses web scraping?

#1

Post by AlexMcP » Wed, Feb 13, 2013, 5:27 AM UTC

A few survey questions about collecting data via the web:

Do you have experience with people using web scraping commercially?

What field or industry was it being used in?

Do you think it fell in a 'grey' area, or was it explicitly disallowed or allowed?

Thanks for any info!

Re: Ask HN: Who uses web scraping?

#2

Post by sheraz » Wed, Feb 13, 2013, 5:43 AM UTC

[1] - Yes

[2] - Lead generation, selling data, consumer + market research

[3] - I think it is always a gray area with regards to another site's TOS. But the burden of proof is on them, not the scraper. That and masking your scrapers is too easy.

Re: Ask HN: Who uses web scraping?

#3

Post by ankitml » Wed, Feb 13, 2013, 5:52 AM UTC

Yes I had done web scrapping commercially, sometime back when I had just left college. I had to scrap website(s) for getting data about all pharma products available in india. So basically market research. But as soon as I realized this was against TOS of the website, i deleted all data and communicated this to client. I had to discard the contract and return even the previous payments. Although I am happy that I did. :)

Re: Ask HN: Who uses web scraping?

#4

Post by ihopngo » Wed, Feb 13, 2013, 1:05 PM UTC

ankitml wrote: post #3

Yes I had done web scrapping commercially, sometime back when I had just left college. I had to scrap website(s) for getting data about all pharma products available in india. So basically market research. But as soon as I realized this was against TOS of the website, i deleted all data and communicated this to client. I had to discard the contract and return even the previous payments. Although I am happy that I did…

Eventbrite used scraping in the early days. They first argued it was a publicly available website. "Public" in the sense that the site was browse-able without restriction.

The plaintiff responded that it had anti-scraping language in its user agreement. The court said "the Terms of Use are not displayed on the website in any way in which a reasonable user could be expected to notice them."