Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Crawl the Web Politely with Scrapy

blog.scrapinghub.com

Post reply on HN
Print view View original

1–10 of 44 posts

Re: How to Crawl the Web Politely with Scrapy

#3

Post by elorant »

In my experience the best way to crawl in a polite way is to never use an asynchronous crawler. The vast majority of small to medium sites out there have absolutely no kind of protection from an aggressive crawler. You make 50 to 100 requests per second chances are you’re DDoS-ing the shit out of most sites.

As for robots.txt problem is most sites don’t even have one. Especially e-commerce sites. They also don’t have a sitemap.xml in case you don’t want to hit every url just to find the structure of the site. Being polite in many cases takes a considerable effort.

Re: How to Crawl the Web Politely with Scrapy

#5

Post by tangue »

Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper.

Instead of fighting against scrapers that don't want to harm you, maybe it's about time to invest in your robots.txt and cooperate.

You could say that scraping you're website is FORBIDDEN, but come on : if Airbnb can rent houses, I can scrap you site.

Re: How to Crawl the Web Politely with Scrapy

#6

Post by jsargiox »

elorant wrote: post #3

In my experience the best way to crawl in a polite way is to never use an asynchronous crawler. The vast majority of small to medium sites out there have absolutely no kind of protection from an aggressive crawler. You make 50 to 100 requests per second chances are you’re DDoS-ing the shit out of most sites. As for robots.txt problem is most sites don’t even have one. Especially e-commerce sites. They also don’t have…

I agree, at the end of the day being polite or not is on the developer and not the tool itself...

Re: How to Crawl the Web Politely with Scrapy

#7

Post by stummjr »

elorant wrote: post #3

In my experience the best way to crawl in a polite way is to never use an asynchronous crawler. The vast majority of small to medium sites out there have absolutely no kind of protection from an aggressive crawler. You make 50 to 100 requests per second chances are you’re DDoS-ing the shit out of most sites. As for robots.txt problem is most sites don’t even have one. Especially e-commerce sites. They also don’t have…

Scrapy is asynchronous, but it provides many settings that you can use to avoid DDoS a website, such as limiting the amount of simultaneous requests for each domain or IP address.

And yes, crawling politely requires a bit of effort from both ends: the crawler and the website.

Re: How to Crawl the Web Politely with Scrapy

#8

Post by vonklaus »

The current protocols promote data exchange and since websites are primarily designed to be consumed, there is really no way to stop automated requests. Even companies like distilli[1] networks that parse inflight requests have trouble stopping any sufficiently motivated outfit.

I think data should be disseminated and free info exchange is great. If possible, devs should respect website owners as much as possible; although in my experience people seem to be more willing to rip off large "faceless" sites rather than mom&&pops. Both because that is where valuable data is, and it seems more justifiable even if morally gray.

Regardless, the thing I find most interesting is that Google is most often criticized for selling user data/out their users privacy. However, it is oft not mentioned that Googlebot & the army of chrome browsers are not only permitted, but encouraged to crawl all sites except a scant few that gave achieved escape velocity. Sites that wish to protect their data must disallow and forcibly stop most crawlers except google, otherwise they will be unranked. This creates an odd dichotomy where not only does google retain massive leverage, but another search engine or aggregator has more hurdles and less resources to compete.

[1] They protect crunchbase and many media companies.

Re: How to Crawl the Web Politely with Scrapy

#9

Post by cookiecaper »

minimaxir wrote: post #2

See also Tuesday's HN discussion on the ethics of data scraping ( https://news.ycombinator.com/item?id=12345952 ), in which Hacker News is completely split on whether data scraping is ethical even if the Terms of Service explicitly forbids it .

I wouldn't say completely split. I think most of HN considers the current state of scraping law to be complete and utter hogwash. Many of the expected consumer rights don't apply online because of the way the law considers normal communication with a server on the internet an excursion onto private property.

We need a modern law addressing these issues instead of the pre-Internet CFAA. Malicious actors should still be punished, and it may be reasonable to still allow a provision for the civil liability (not criminal) of large-scale accidental DoS from poorly-implemented scrapers, but users should be free to choose their own browsing devices -- even if those browsing devices are highly optimized to extract only the specific pieces of data that the user cares about.

This law should also clarify that normal communication over HTTP cannot be punished unless the plaintiff can demonstrate real and serious interruption to their services, that local RAM copies that are never externally transmitted cannot be considered infringing in themselves, that hosting a site on the internet grants an implied copyright license to read and access its content with any HTTP-capable client, and that browsewrap/clickwrap contracts are unenforceable unless the user undertakes a significant relationship with the company, among other things.

Re: How to Crawl the Web Politely with Scrapy

#10

Post by emodendroket »

minimaxir wrote: post #2

See also Tuesday's HN discussion on the ethics of data scraping ( https://news.ycombinator.com/item?id=12345952 ), in which Hacker News is completely split on whether data scraping is ethical even if the Terms of Service explicitly forbids it .

Are you trying to imply that's a ridiculous position? I don't see it as one.
Post reply on HN