Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Crawl the Web Politely with Scrapy

blog.scrapinghub.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: How to Crawl the Web Politely with Scrapy

#21

Post by AznHisoka »

betolink wrote: post #20

I worked on a research project to develop a web-scale "google" for scientific data and we found very interesting things on robots.txt, from "don't crawl us" to "crawl 1 page every other day" or even better "don't crawl unless you're google". Another thing we noticed is that google's crawler is kind of aggressive, I guess they are in a position to do it. Our paper in case someone is interested: Optimizing Apache Nutch…

This is why I think Google's position as the #1 search engine will never go away. Many sites will tell your bot to go away if you're not Google. They don't care if you're building a search engine that will compete with Google.

Re: How to Crawl the Web Politely with Scrapy

#22

Post by markpapadakis »

In the past we built and operated Greece’s largest search engine(Trinity), and we would crawl/refresh all Greek pages fairly regularly.

If memory serves, the frequency was computed for clusters of pages from the same site, and it depended on how often they were updated(news sites front-pages were in practice different in successive updates, whereas e.g users homepage were not, they rarely were updated), and how resilient the sites were to aggressive indexing (if they ‘d fail or timeout, or it ‘d take longer than expected to download the page contents than what we expected based on site-wide aggregated metrics, we ‘d adjust the frequency, etc).

The crawlers were all draining multiple queues, whereas URLs from the same site would always end up on the same queue(via consistent hashing, based on the hostname’s hash), so a single crawler process was responsible for throttling requests and respecting robots.txt rules for any single site, without need for cross-crawler state synchronisation.

In practice this worked quite well. Also, this was before Google and its PageRank and social networks (we ‘d probably have also considered pages popularity based on PageRank like metrics and social ‘signals’ in the frequency computation, among other variables).

Re: How to Crawl the Web Politely with Scrapy

#23

Post by greglindahl »

AznHisoka wrote: post #21
betolink wrote: post #20

I worked on a research project to develop a web-scale "google" for scientific data and we found very interesting things on robots.txt, from "don't crawl us" to "crawl 1 page every other day" or even better "don't crawl unless you're google". Another thing we noticed is that google's crawler is kind of aggressive, I guess they are in a position to do it. Our paper in case someone is interested: Optimizing Apache Nutch…

This is why I think Google's position as the #1 search engine will never go away. Many sites will tell your bot to go away if you're not Google. They don't care if you're building a search engine that will compete with Google.

At blekko, we did not find this issue to be a significant one... almost everyone who banned our crawler was a crappy over-SEOed website.

Re: How to Crawl the Web Politely with Scrapy

#24

Post by emodendroket »

minimaxir wrote: post #12
emodendroket wrote: post #10

Earlier quoted context omitted.

Are you trying to imply that's a ridiculous position? I don't see it as one.

In the context of this submission, it's definitely not polite .

It's certainly more considerate if you limit your scraping to rates similar to a human user instead of just going as fast as possible.

Re: How to Crawl the Web Politely with Scrapy

#25

Post by atmosx »

markpapadakis wrote: post #22

In the past we built and operated Greece’s largest search engine(Trinity), and we would crawl/refresh all Greek pages fairly regularly. If memory serves, the frequency was computed for clusters of pages from the same site, and it depended on how often they were updated(news sites front-pages were in practice different in successive updates, whereas e.g users homepage were not, they rarely were updated), and how resil…

Hi Mark, out of curiosity, which search engine is that?

Re: How to Crawl the Web Politely with Scrapy

#26

Post by markpapadakis »

atmosx wrote: post #25
markpapadakis wrote: post #22

In the past we built and operated Greece’s largest search engine(Trinity), and we would crawl/refresh all Greek pages fairly regularly. If memory serves, the frequency was computed for clusters of pages from the same site, and it depended on how often they were updated(news sites front-pages were in practice different in successive updates, whereas e.g users homepage were not, they rarely were updated), and how resil…

Hi Mark, out of curiosity, which search engine is that?

It was called Trinity -- it was initially developed for Pathfinder.gr, and soon thereafter was the search provider for in.gr, and was also accessible at trinity.gr for some time.

Re: How to Crawl the Web Politely with Scrapy

#27

Post by pmoriarty »

cookiecaper wrote: post #11
tangue wrote: post #5

Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper. Instead of fighting against scrapers that don't want to harm you, maybe it's about time to invest in your robots.txt and cooperate. You could say that scraping you're website is FORBIDDEN, b…

>Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper. Unfortunately, if you're scraping some data that only has one authoritative data source, they'll know you're scraping them even if they can't distinguish your individual requests from the g…

"Unfortunately, if you're scraping some data that only has one authoritative data source, they'll know you're scraping them even if they can't distinguish your individual requests from the general traffic."

How would they know you're scraping them?

Surely the capability of any given website admin to detect a particular scraper would depend on many factors such as whether they're even looking for scrapers or are technologically capable of doing so, how many/which IPs the scraping is originating from, and how cleverly the scraper goes about their scraping, no?

It's a bit of a cat and mouse game, wouldn't you say?

Re: How to Crawl the Web Politely with Scrapy

#28

Post by greglindahl »

markpapadakis wrote: post #22

In the past we built and operated Greece’s largest search engine(Trinity), and we would crawl/refresh all Greek pages fairly regularly. If memory serves, the frequency was computed for clusters of pages from the same site, and it depended on how often they were updated(news sites front-pages were in practice different in successive updates, whereas e.g users homepage were not, they rarely were updated), and how resil…

In the current web, sites like Amazon are so large that you'll need many crawlers. On the plus side, it appears that almost all large sites don't have rate limits.

Re: How to Crawl the Web Politely with Scrapy

#29

Post by cookiecaper »

pmoriarty wrote: post #27
cookiecaper wrote: post #11

Earlier quoted context omitted.

>Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper. Unfortunately, if you're scraping some data that only has one authoritative data source, they'll know you're scraping them even if they can't distinguish your individual requests from the g…

"Unfortunately, if you're scraping some data that only has one authoritative data source, they'll know you're scraping them even if they can't distinguish your individual requests from the general traffic." How would they know you're scraping them? Surely the capability of any given website admin to detect a particular scraper would depend on many factors such as whether they're even looking for scrapers or are techn…

They know you're scraping them because their site is the only source of the data you're scraping. The most common example here is airlines. Airlines that haven't agreed to be included in fare aggregators often have their booking information scraped. Even if your traffic blends in, they know that you're reading out fare data from them, because where else would you get it from? This is especially true if you follow it up with a link to buy the specific fare at the airline's site. The only plausible way to have that is to read it off of their site (and, even if you can use a template based on their URL structure, I think there would probably be a case to be made that URLs qualify for copyright and trademark protection).

As for the game of cat and mouse, it lasts until they call in their lawyers. Then it's a game of "quit now or get destroyed".

But yes, if you can scrape the data without ever tipping off the company you're scraping, you can probably continue indefinitely, but you have to consider whether you can plausibly argue that you're getting that data from someplace else. If they sued you on the suspicion that you're scraping them, they'll probably subpoena the code to confirm that (or similar -- IANAL), and then proceed to try to make a case on things other than CFAA violations.

Re: How to Crawl the Web Politely with Scrapy

#30

Post by stummjr »

greglindahl wrote: post #28
markpapadakis wrote: post #22

In the past we built and operated Greece’s largest search engine(Trinity), and we would crawl/refresh all Greek pages fairly regularly. If memory serves, the frequency was computed for clusters of pages from the same site, and it depended on how often they were updated(news sites front-pages were in practice different in successive updates, whereas e.g users homepage were not, they rarely were updated), and how resil…

In the current web, sites like Amazon are so large that you'll need many crawlers. On the plus side, it appears that almost all large sites don't have rate limits.

Crawl-delay is not in the standard robots.txt protocol, and according to Wikipedia, some bots have different interpretations for this value. That's why maybe many websites don't even bother defining the rate limits in robots.txt.
Post reply on HN