Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrapy Tips from the Pros

blog.scrapinghub.com

Post reply on HN
Print view View original

11–20 of 78 posts

Re: Scrapy Tips from the Pros

#11

Post by mryan »

stummjr wrote: post #2

Hey, author here! Feel free to ask any questions you have.

Great article, thanks for sharing.

What would you recommend for people who need to operate a cluster of scrapyd instances? Are there any recommended ways of managing the distribution of tasks to multiple scrapyd instances?

I have looked at scrapyd-cluster, but I would prefer not to add a Zookeeper cluster to my stack. Currently I'm thinking of modifying scrapyd-cluster so that it uses AMQP (via Celery) to handle task distribution/retries/etc.

I appreciate this conflicts with the scrapinghub business model, so any tips you can offer would be greatly appreciated :-)

Re: Scrapy Tips from the Pros

#12

Post by rahulrrixe »

I have been using this framework for more than three years and seen how it has evolved and made scrapping so easy. The portia project is also awesome. I have customised scrapy for almost all the cases like having single spiders for multiple sites and providing rules using JSON. I think it is highly scalable with bit of tweak and scrapy allows you to do very easily.

Re: Scrapy Tips from the Pros

#13

Post by mitchtbaum »

stummjr wrote: post #7
piroux wrote: post #5

Earlier quoted context omitted.

Here is a first one : What are the best ways to detect changes in html sources with scrapy, thus giving missing data in automatic systems that need to be fed ?

Hey, not sure if I understood what you mean. Did you mean: 1) detect pages that had changed since the last crawl, to avoid recrawling pages that hadn't changed? 2) detect pages that have changed their structure, breaking down the Spider that crawl it.

> 1) detect pages that had changed since the last crawl, to avoid recrawling pages that hadn't changed?

Usually web clients use https://en.wikipedia.org/wiki/HTTP_ETag , afais. If a web app\server lacks that skill, then you could compute your own hash and check it yourself, instead of processing that condition at the network layer.

Re: Scrapy Tips from the Pros

#14

Post by horva »

Cyph0n wrote: post #8

Extremely well designed framework. It can cover more than 90% of use cases in my opinion. I'm currently working on a project written in Scala that requires a lot of scraping, and I feel really guilty that I'm not using Scrapy :(

I think you can still combine the two. For example Scrapy can be behind service/server to which you'd send request (with same args as if you were running it as a script + callback url) and after items get collected Scrapy can call your callback url sending all items in json format to your Scala app. Or if you want to avoid memory issues for sure, you can send each item to Scala app as it gets collected. Basically, idea is to wrap Scrapy spiders with web service features - then you can use it in combination with any other technology. Or you can use Scrapy Cloud to run your spiders at http://scrapinghub.com/.

Re: Scrapy Tips from the Pros

#15

Post by khgvljhkb »

Cyph0n wrote: post #9
nathell wrote: post #3

Wow, never heard of Scrapy! Looks like I've reinvented it in Clojure: https://github.com/nathell/skyscraper/

That looks pretty cool. I was planning on writing something similar in Scala, but I'm not sure if I have enough experience with the language to get it done.

If you're lazy (and if you're into FP you must be hehehe), just use that Clojure library. Calling Clojure code from Java is easy, and I'm sure it's not much harder from Scala.

Re: Scrapy Tips from the Pros

#16

Post by inovica »

We use Scrapy for a few projects and it is really really good. They have a commercial-side to them, which is fine, but for anyone doing crawling/scraping I'd strongly recommend it. Good article also

Re: Scrapy Tips from the Pros

#17

Post by IanCal »

horva wrote: post #14
Cyph0n wrote: post #8

Extremely well designed framework. It can cover more than 90% of use cases in my opinion. I'm currently working on a project written in Scala that requires a lot of scraping, and I feel really guilty that I'm not using Scrapy :(

I think you can still combine the two. For example Scrapy can be behind service/server to which you'd send request (with same args as if you were running it as a script + callback url) and after items get collected Scrapy can call your callback url sending all items in json format to your Scala app. Or if you want to avoid memory issues for sure, you can send each item to Scala app as it gets collected. Basically, id…

> Or if you want to avoid memory issues for sure, you can send each item to Scala app as it gets collected.

I've done something similar, you can just add a Pipeline (Pipeline Task? not sure of the right terminology) which posts off the data somewhere else. You can also then store the full item logs in S3 so you can go back occasionally and check nothing has been missed. Works an absolute treat.

Re: Scrapy Tips from the Pros

#18

Post by IanCal »

inovica wrote: post #16

We use Scrapy for a few projects and it is really really good. They have a commercial-side to them, which is fine, but for anyone doing crawling/scraping I'd strongly recommend it. Good article also

Same, it's really nicely put together, lots of sensible defaults and it's easy to add your own bit of awkward logic when necessary.

Re: Scrapy Tips from the Pros

#19

Post by stummjr »

stummjr wrote: post #7
piroux wrote: post #5

Earlier quoted context omitted.

Here is a first one : What are the best ways to detect changes in html sources with scrapy, thus giving missing data in automatic systems that need to be fed ?

Hey, not sure if I understood what you mean. Did you mean: 1) detect pages that had changed since the last crawl, to avoid recrawling pages that hadn't changed? 2) detect pages that have changed their structure, breaking down the Spider that crawl it.

1) detect pages that had changed since the last crawl, to avoid recrawling pages that hadn't changed?

You could use the deltafetch[1] middleware. It ignores requests to pages with items extracted in previous crawls.

2) detect pages that have changed their structure, breaking down the Spider that crawl it.

This is a tough one, since most of the spiders are heavily based on the HTML structure. You could use Spidermon [2] to monitor your spiders. It's available as an addon in the Scrapy Cloud platform [3], and there are plans to open source it in the near future. Also, dealing automatically with pages that change their structure is in the roadmap for Portia [4].

[1] https://github.com/scrapinghub/scrapylib/blob/master/scrapyl...

[2] http://doc.scrapinghub.com/addons.html?highlight=monitoring#...

[3] http://scrapinghub.com/scrapy-cloud/

[4] http://scrapinghub.com/portia/

Re: Scrapy Tips from the Pros

#20

Post by stummjr »

Cyph0n wrote: post #8

Extremely well designed framework. It can cover more than 90% of use cases in my opinion. I'm currently working on a project written in Scala that requires a lot of scraping, and I feel really guilty that I'm not using Scrapy :(

There is an idea to make Scrapy support Spiders written in other languages[1]. It has been featured on the Google Summer of Code 2015[2].

[1] https://github.com/scrapy/scrapy/issues/1125

[2] http://gsoc2015.scrapinghub.com/ideas/#other-languages

Post reply on HN