Scrapy Tips from the Pros • Hacker News Board

Re: Scrapy Tips from the Pros

#61

Post by shostack » Wed, Jan 20, 2016, 9:31 PM UTC

ddebernardy wrote: post #49

JoshTriplett wrote: post #47

Earlier quoted context omitted.

Crawling sites actively hostile to your crawler seems like it has both useful and shady applications. What kinds of things do you use that for?

On the one hand side there's no shortage of users who want to crawl popular sites to monitor e.g. search engine ranking or prices. Which is kind of shady in some sense, or not - when there's no API there's no other way... On the other there are also areas of the web where crawlers are simply not welcome. For instance, DARPA uses a number of our technologies to monitor the dark web for criminal activities: http://open…

As an "early" programmer playing with web scraping with the Nokogiri gem, I've been wondering about this aspect (although haven't encountered it yet).

Are there legal implications to scraping a site that actively tries to prevent bots from scraping it? I mean, if the data is publicly accessible on the web, could they go after you?

I don't plan on doing this for any malicious reasons or anything, and like I said, I haven't encountered it yet. Just having the "what if" thought of what my legal risks might be if I'm playing around with this and whether a site could come after me.

Re: Scrapy Tips from the Pros

#62

Post by phunge » Wed, Jan 20, 2016, 9:32 PM UTC

kami8845 wrote: post #29

I love ScrapingHub (and use them) but these tips go completely against my own experience. Whenever I've tried to extract data like that inside Spiders I would invariably (and 50,000 URLs later) come to the realization that my .parse() ing code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. How to re-run all that with more robust lo…

Preach! :)

That's my preferred design too. The first job of any external data capture process is to capture the full fidelity source data. Everything else belongs in a followon job.

Re: Scrapy Tips from the Pros

#63

Post by shostack » Wed, Jan 20, 2016, 9:35 PM UTC

Rubyist here...how does Scrapy compare to Nokogiri?

Re: Scrapy Tips from the Pros

#64

Post by ddebernardy » Wed, Jan 20, 2016, 10:09 PM UTC

shostack wrote: post #61

ddebernardy wrote: post #49

Earlier quoted context omitted.

On the one hand side there's no shortage of users who want to crawl popular sites to monitor e.g. search engine ranking or prices. Which is kind of shady in some sense, or not - when there's no API there's no other way... On the other there are also areas of the web where crawlers are simply not welcome. For instance, DARPA uses a number of our technologies to monitor the dark web for criminal activities: http://open…

As an "early" programmer playing with web scraping with the Nokogiri gem, I've been wondering about this aspect (although haven't encountered it yet). Are there legal implications to scraping a site that actively tries to prevent bots from scraping it? I mean, if the data is publicly accessible on the web, could they go after you? I don't plan on doing this for any malicious reasons or anything, and like I said, I ha…

> Are there legal implications to scraping a site that actively tries to prevent bots from scraping it? I mean, if the data is publicly accessible on the web, could they go after you?

When we do projects, the baseline is if Google can see it we can too. So from a legal standpoint if Google is covered so are we.

From a legal standpoint firms do go after web scrapers. And lose more often than not. The exception is when you're logged in when you crawl. In that case you've implicitly accepted the terms of use. Some companies aggressively sue when you're logged in while scraping, so it's best to stay on the safe side. Further reading on the topic:

https://www.quora.com/What-is-the-legality-of-web-scraping

Re: Scrapy Tips from the Pros

#65

Post by ddebernardy » Wed, Jan 20, 2016, 10:13 PM UTC

staticautomatic wrote: post #57

I like this article but for its discussion of these libraries. On another note... Am I the only one who dislikes Scrapy? I think it's basically the iOS of scraping tools: It's incredibly easy to setup and use, and then as soon as you need to do something even minutely non-standard it reveals itself to be frustratingly inflexible.

Scrapy is about as flexible and extensible as you can get... Care to elaborate on "frustratingly inflexible"?

Re: Scrapy Tips from the Pros

#66

Post by rcfox » Wed, Jan 20, 2016, 10:19 PM UTC

kami8845 wrote: post #29

I love ScrapingHub (and use them) but these tips go completely against my own experience. Whenever I've tried to extract data like that inside Spiders I would invariably (and 50,000 URLs later) come to the realization that my .parse() ing code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. How to re-run all that with more robust lo…

The very first thing I do with every scraping project is enable the HttpCacheMiddleware[0]. After downloading a page once, subsequent runs will automatically pull it from the local cache. This makes it way faster to experiment, and doesn't increase the burden of the website.

[0] http://doc.scrapy.org/en/latest/topics/downloader-middleware...

Re: Scrapy Tips from the Pros

#67

Post by ddebernardy » Wed, Jan 20, 2016, 10:24 PM UTC

shostack wrote: post #63

Rubyist here...how does Scrapy compare to Nokogiri?

Much like apples to oranges.

Nokogiri is a tag soup parser. Scrapy is a web scraping framework.

In addition to tag soup parsing, Scrapy handles a slew of things such as text encoding problems, retrying urls that fail owing to network problems (if you wish), dispatching requests across multiple spiders with a shared crawl frontier (see Frontera), shared code between similar spiders using middlewares and pipelines, and what have you.

There's a Lisp joke that goes something like every sufficiently complex piece of software in C is a slow, buggy, poorly implemented version of Lisp. Very much the same could be said about Scrapy and web scraping projects. :-)

Re: Scrapy Tips from the Pros

#68

Post by shostack » Wed, Jan 20, 2016, 10:32 PM UTC

ddebernardy wrote: post #67

shostack wrote: post #63

Rubyist here...how does Scrapy compare to Nokogiri?

Much like apples to oranges. Nokogiri is a tag soup parser. Scrapy is a web scraping framework. In addition to tag soup parsing, Scrapy handles a slew of things such as text encoding problems, retrying urls that fail owing to network problems (if you wish), dispatching requests across multiple spiders with a shared crawl frontier (see Frontera), shared code between similar spiders using middlewares and pipelines, and…

Makes sense, thanks for the clarification. To your knowledge, is there anything comparable to Scrapy in Ruby land?

Re: Scrapy Tips from the Pros

#69

Post by elktea » Wed, Jan 20, 2016, 10:56 PM UTC

Shame Python 3 isn't there for Scrapy yet - although checking the repo looks like someone is actively (last few hours) porting it. Good to see.

Re: Scrapy Tips from the Pros

#70

Post by binarysolo » Thu, Jan 21, 2016, 1:30 AM UTC

kami8845 wrote: post #29

I love ScrapingHub (and use them) but these tips go completely against my own experience. Whenever I've tried to extract data like that inside Spiders I would invariably (and 50,000 URLs later) come to the realization that my .parse() ing code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. How to re-run all that with more robust lo…

Seconding the practice of collect, then parse.

Storage is cheap relative to hammering a server (or working around the bounds of rate-limitation), so always save your collected things, then crawl it as many time as you desire as you play around with the right scrapes.