Scrapy Tips from the Pros

#1

Post by ddebernardy » Wed, Jan 20, 2016, 10:25 AM UTC

Scrapy Tips from the Pros
blog.scrapinghub.com

Re: Scrapy Tips from the Pros

#2

Post by stummjr » Wed, Jan 20, 2016, 11:24 AM UTC

Hey, author here! Feel free to ask any questions you have.

Re: Scrapy Tips from the Pros

#3

Post by nathell » Wed, Jan 20, 2016, 11:28 AM UTC

Wow, never heard of Scrapy! Looks like I've reinvented it in Clojure: https://github.com/nathell/skyscraper/

Re: Scrapy Tips from the Pros

#4

Post by daturkel » Wed, Jan 20, 2016, 11:35 AM UTC

I've played with Scrapy before to make a proof of concept and I was pleased with how easy it was (haven't yet had to use it for anything else yet).

That being said, had no idea how sophisticated it could get. This is super impressive, especially the JavaScript rendering.

Re: Scrapy Tips from the Pros

#5

Post by piroux » Wed, Jan 20, 2016, 11:37 AM UTC

stummjr wrote: post #2

Hey, author here! Feel free to ask any questions you have.

Here is a first one : What are the best ways to detect changes in html sources with scrapy, thus giving missing data in automatic systems that need to be fed ?

Re: Scrapy Tips from the Pros

#6

Post by [deleted] » Wed, Jan 20, 2016, 11:55 AM UTC

piroux wrote: post #5

stummjr wrote: post #2

Hey, author here! Feel free to ask any questions you have.

Here is a first one : What are the best ways to detect changes in html sources with scrapy, thus giving missing data in automatic systems that need to be fed ?

[deleted]

Re: Scrapy Tips from the Pros

#7

Post by stummjr » Wed, Jan 20, 2016, 11:58 AM UTC

piroux wrote: post #5

stummjr wrote: post #2

Hey, author here! Feel free to ask any questions you have.

Here is a first one : What are the best ways to detect changes in html sources with scrapy, thus giving missing data in automatic systems that need to be fed ?

Hey, not sure if I understood what you mean. Did you mean:

1) detect pages that had changed since the last crawl, to avoid recrawling pages that hadn't changed? 2) detect pages that have changed their structure, breaking down the Spider that crawl it.

Re: Scrapy Tips from the Pros

#8

Post by Cyph0n » Wed, Jan 20, 2016, 12:02 PM UTC

Extremely well designed framework. It can cover more than 90% of use cases in my opinion. I'm currently working on a project written in Scala that requires a lot of scraping, and I feel really guilty that I'm not using Scrapy :(

Re: Scrapy Tips from the Pros

#9

Post by Cyph0n » Wed, Jan 20, 2016, 12:04 PM UTC

nathell wrote: post #3

Wow, never heard of Scrapy! Looks like I've reinvented it in Clojure: https://github.com/nathell/skyscraper/

That looks pretty cool. I was planning on writing something similar in Scala, but I'm not sure if I have enough experience with the language to get it done.

Re: Scrapy Tips from the Pros

#10

Post by grantbachman » Wed, Jan 20, 2016, 12:10 PM UTC

stummjr wrote: post #7

piroux wrote: post #5

Earlier quoted context omitted.

Here is a first one : What are the best ways to detect changes in html sources with scrapy, thus giving missing data in automatic systems that need to be fed ?

Hey, not sure if I understood what you mean. Did you mean: 1) detect pages that had changed since the last crawl, to avoid recrawling pages that hadn't changed? 2) detect pages that have changed their structure, breaking down the Spider that crawl it.

As someone who does a fair amount of scraping at his job, I'd like to hear what you have to say regarding both questions :)