How to Crawl the Web Politely with Scrapy • Hacker News Board

Re: How to Crawl the Web Politely with Scrapy

#11

Post by cookiecaper » Thu, Aug 25, 2016, 3:20 PM UTC

tangue wrote: post #5

Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper. Instead of fighting against scrapers that don't want to harm you, maybe it's about time to invest in your robots.txt and cooperate. You could say that scraping you're website is FORBIDDEN, b…

>Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper.

Unfortunately, if you're scraping some data that only has one authoritative data source, they'll know you're scraping them even if they can't distinguish your individual requests from the general traffic.

This is what happened to my company. It didn't stop them from pretending that we were setting their servers on fire, even though they had no way to know whether we were or not since they couldn't distinguish our traffic from that generated by other browsers.

We were scraping only factual data in the which the company cannot hold a copyright interest. Nonetheless, under Ticketmaster v. RMG, just holding a copy of a page in RAM long enough to parse it constitutes infringement (you have to prove fair use, as Google supposedly did in Perfect 10 v. Google, to avoid this).

The difference between yourself and Google/airbnb is that the latter have a lot of money and are trendy technology companies, and you don't and aren't (yet).

The lesson is become really big before someone sues you and the judiciary will be on your side.

Re: How to Crawl the Web Politely with Scrapy

#12

Post by minimaxir » Thu, Aug 25, 2016, 3:23 PM UTC

emodendroket wrote: post #10

minimaxir wrote: post #2

See also Tuesday's HN discussion on the ethics of data scraping ( https://news.ycombinator.com/item?id=12345952 ), in which Hacker News is completely split on whether data scraping is ethical even if the Terms of Service explicitly forbids it .

Are you trying to imply that's a ridiculous position? I don't see it as one.

In the context of this submission, it's definitely not polite.

Re: How to Crawl the Web Politely with Scrapy

#13

Post by FussyZeus » Thu, Aug 25, 2016, 3:24 PM UTC

tangue wrote: post #5

Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper. Instead of fighting against scrapers that don't want to harm you, maybe it's about time to invest in your robots.txt and cooperate. You could say that scraping you're website is FORBIDDEN, b…

It depends on your definition of harm. When your product is what's published on the websites and you regularly find ripoffs of said website publishing your ripped off content, maybe you'd feel differently about it.

Re: How to Crawl the Web Politely with Scrapy

#14

Post by duiker101 » Thu, Aug 25, 2016, 3:49 PM UTC

FussyZeus wrote: post #13

tangue wrote: post #5

Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper. Instead of fighting against scrapers that don't want to harm you, maybe it's about time to invest in your robots.txt and cooperate. You could say that scraping you're website is FORBIDDEN, b…

It depends on your definition of harm. When your product is what's published on the websites and you regularly find ripoffs of said website publishing your ripped off content, maybe you'd feel differently about it.

fair enough but I don't think that's the main purpose. There are many many cases where you would want to scrape something and often people would probably be encouraged in doing so in a "polite" way if websites didn't make it hard.

Re: How to Crawl the Web Politely with Scrapy

#15

Post by zeroxfe » Thu, Aug 25, 2016, 3:57 PM UTC

elorant wrote: post #3

In my experience the best way to crawl in a polite way is to never use an asynchronous crawler. The vast majority of small to medium sites out there have absolutely no kind of protection from an aggressive crawler. You make 50 to 100 requests per second chances are you’re DDoS-ing the shit out of most sites. As for robots.txt problem is most sites don’t even have one. Especially e-commerce sites. They also don’t have…

You can rate-limit asynchronous crawlers too.

Re: How to Crawl the Web Politely with Scrapy

#16

Post by stummjr » Thu, Aug 25, 2016, 4:03 PM UTC

FussyZeus wrote: post #13

tangue wrote: post #5

Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper. Instead of fighting against scrapers that don't want to harm you, maybe it's about time to invest in your robots.txt and cooperate. You could say that scraping you're website is FORBIDDEN, b…

It depends on your definition of harm. When your product is what's published on the websites and you regularly find ripoffs of said website publishing your ripped off content, maybe you'd feel differently about it.

Yeah, but that's not just because of web scraping. Plagiarism has been an issue for centuries.

Re: How to Crawl the Web Politely with Scrapy

#17

Post by greglindahl » Thu, Aug 25, 2016, 4:15 PM UTC

elorant wrote: post #3

In my experience the best way to crawl in a polite way is to never use an asynchronous crawler. The vast majority of small to medium sites out there have absolutely no kind of protection from an aggressive crawler. You make 50 to 100 requests per second chances are you’re DDoS-ing the shit out of most sites. As for robots.txt problem is most sites don’t even have one. Especially e-commerce sites. They also don’t have…

Search engine crawlers use adaptive politeness: start being very polite, and ramp up parallel fetches if the site responds quickly and has a lot of pages.

Re: How to Crawl the Web Politely with Scrapy

#18

Post by stummjr » Thu, Aug 25, 2016, 4:30 PM UTC

greglindahl wrote: post #17

elorant wrote: post #3

In my experience the best way to crawl in a polite way is to never use an asynchronous crawler. The vast majority of small to medium sites out there have absolutely no kind of protection from an aggressive crawler. You make 50 to 100 requests per second chances are you’re DDoS-ing the shit out of most sites. As for robots.txt problem is most sites don’t even have one. Especially e-commerce sites. They also don’t have…

Search engine crawlers use adaptive politeness: start being very polite, and ramp up parallel fetches if the site responds quickly and has a lot of pages.

That's kind of what Scrapy's AUTO_THROTTLE middleware does.

Re: How to Crawl the Web Politely with Scrapy

#19

Post by libeclipse » Thu, Aug 25, 2016, 4:55 PM UTC

If you're worried about being a pain in the ass to administrators, with a web-scraper, they probably need to rethink the way they have their website set up.

Re: How to Crawl the Web Politely with Scrapy

#20

Post by betolink » Thu, Aug 25, 2016, 5:07 PM UTC

I worked on a research project to develop a web-scale "google" for scientific data and we found very interesting things on robots.txt, from "don't crawl us" to "crawl 1 page every other day" or even better "don't crawl unless you're google".

Another thing we noticed is that google's crawler is kind of aggressive, I guess they are in a position to do it.

Our paper in case someone is interested: Optimizing Apache Nutch for domain specific crawling at large scale (http://ieeexplore.ieee.org/document/7363976/?arnumber=736397...)