Live data from Hacker News

How to Crawl the Web Politely with Scrapy

blog.scrapinghub.com

31–40 of 44 posts

Re: How to Crawl the Web Politely with Scrapy

#31

Earlier quoted context omitted.

"Unfortunately, if you're scraping some data that only has one authoritative data source, they'll know you're scraping them even if they can't distinguish your individual requests from the general traffic." How would they know you're scraping them? Surely the capability of any given website admin to detect a particular scraper would depend on many factors such as whether they're even looking for scrapers or are techn…

They know you're scraping them because their site is the only source of the data you're scraping. The most common example here is airlines. Airlines that haven't agreed to be included in fare aggregators often have their booking information scraped. Even if your traffic blends in, they know that you're reading out fare data from them, because where else would you get it from? This is especially true if you follow it…

Oh, you're talking about them inferring that you must have scraped them because you used or published data that only they had.

Not every scraper has publishing or using data in a detectable way as their motive.

For instance, I sometimes scrape a website to make an archive of it for my own personal use. I never publish the results or use them in any way that the website owners would ever know about. So the only way they could know that they were scraped is if I left some kind of scraping signature while scraping (such as scraping from a single IP and doing it quick enough to pop on their radar or perhaps regularly enough -- ie. without random waits between request, etc).

What you're talking about is probably mostly a concern to people/companies who are somehow making money from scraping data on other people's websites.

Re: How to Crawl the Web Politely with Scrapy

#32
post #30

Earlier quoted context omitted.

In the current web, sites like Amazon are so large that you'll need many crawlers. On the plus side, it appears that almost all large sites don't have rate limits.

Crawl-delay is not in the standard robots.txt protocol, and according to Wikipedia, some bots have different interpretations for this value. That's why maybe many websites don't even bother defining the rate limits in robots.txt.

I was referring to an actual rate limit, not crawl-delay. For example, YouTube is pretty strict about rate limits:

http://www.bing.com/search?q=%22We+have+been+receiving+a+lar...

I agree that crawl-delay is rare, and often it's set too long so that it's impossible to fully crawl a site -- as if the webmaster set it up 10 years ago and never updated it as their site got faster and bigger.

Re: How to Crawl the Web Politely with Scrapy

#33
post #5

Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper. Instead of fighting against scrapers that don't want to harm you, maybe it's about time to invest in your robots.txt and cooperate. You could say that scraping you're website is FORBIDDEN, b…

It depends on your definition of harm. When your product is what's published on the websites and you regularly find ripoffs of said website publishing your ripped off content, maybe you'd feel differently about it.

Not sure what that has to do with scraping. A desktop browser can be used to copy and paste chunks of content and plagiarize a site. We have reasonable copyright protections to protect authors against that. What we need to discharge are the unreasonable laws regarding network access. It's not all or nothing.

Re: How to Crawl the Web Politely with Scrapy

#34

See also Tuesday's HN discussion on the ethics of data scraping ( https://news.ycombinator.com/item?id=12345952 ), in which Hacker News is completely split on whether data scraping is ethical even if the Terms of Service explicitly forbids it .

Are you trying to imply that's a ridiculous position? I don't see it as one.

I'll say it's a ridiculous position. How can it possibly be ethical? The owner of the server and the content has specifically told you to stop sending packets at it.

I honestly don't know how to construct an argument for this because it's so obvious to me.

Re: How to Crawl the Web Politely with Scrapy

#35

Earlier quoted context omitted.

This is why I think Google's position as the #1 search engine will never go away. Many sites will tell your bot to go away if you're not Google. They don't care if you're building a search engine that will compete with Google.

At blekko, we did not find this issue to be a significant one... almost everyone who banned our crawler was a crappy over-SEOed website.

https://www.linkedin.com/robots.txt

https://yelp.com/robots.txt

There goes all Linkedin + Yelp content from your index.

Re: How to Crawl the Web Politely with Scrapy

#36

Earlier quoted context omitted.

It depends on your definition of harm. When your product is what's published on the websites and you regularly find ripoffs of said website publishing your ripped off content, maybe you'd feel differently about it.

fair enough but I don't think that's the main purpose. There are many many cases where you would want to scrape something and often people would probably be encouraged in doing so in a "polite" way if websites didn't make it hard.

Yes, or if they just provided a csv with all the data most people wanted to scrape anyway with a plain English explanation about how it can be used.

Re: How to Crawl the Web Politely with Scrapy

#38
post #5

Reading the previous thread again, I suppose that many of those against scraping didn't realized they've already lost : with Ghost, Phantom, and now headless Chrome you're going to have a hard time to detect a well built scraper. Instead of fighting against scrapers that don't want to harm you, maybe it's about time to invest in your robots.txt and cooperate. You could say that scraping you're website is FORBIDDEN, b…

It depends on your definition of harm. When your product is what's published on the websites and you regularly find ripoffs of said website publishing your ripped off content, maybe you'd feel differently about it.

That argument only holds up if you believe in intellectual property. Many of us here do not.

Re: How to Crawl the Web Politely with Scrapy

#39

Earlier quoted context omitted.

At blekko, we did not find this issue to be a significant one... almost everyone who banned our crawler was a crappy over-SEOed website.

https://www.linkedin.com/robots.txt https://yelp.com/robots.txt There goes all Linkedin + Yelp content from your index.

[deleted]

Re: How to Crawl the Web Politely with Scrapy

#40

Earlier quoted context omitted.

At blekko, we did not find this issue to be a significant one... almost everyone who banned our crawler was a crappy over-SEOed website.

https://www.linkedin.com/robots.txt https://yelp.com/robots.txt There goes all Linkedin + Yelp content from your index.

What about https://www.facebook.com/robots.txt

..and medium-sized/small sites are even worse.

The irony of Facebook being a core part of all NSA surveillance programs and their terms of service including their "Automated Data Collection Terms" https://www.facebook.com/apps/site_scraping_tos_terms.php

Post reply on HN