Scrapy Tips from the Pros • Hacker News Board

Re: Scrapy Tips from the Pros

#31

Post by ddebernardy » Wed, Jan 20, 2016, 2:01 PM UTC

hokkos wrote: post #25

What I need is an API scraper, scrapy seems to be mostly for HTML. I know how to look at network requests in Chrome dev tools and JS function to understand the shape of REST API, so I need something to plan the exploration of the arguments space. For example if you want to scrap airbnb, you look at their API, find there is a REST call with a lat, long box, I need something to automatically explore an area, if the api…

It's not entirely clear what you're up to, but FYI Scrapy works fine when scraping JSON data:

http://stackoverflow.com/a/18172776/417194

Re: Scrapy Tips from the Pros

#32

Post by melling » Wed, Jan 20, 2016, 2:05 PM UTC

contingencies wrote: post #24

Man, I feel old. Does anyone remember learning web scraping from one section of Fravia's site? Ever try to move forward from that to write a fully fledged search engine? These memories are from 15 years ago... quite amusing how much hasn't changed. In hindsight it was probably easier back then due to the lack of JS-reliant pages, less awareness of automation and less scraper detection algorithms.

What is the state of web scraping? I've got a few thousand URLs that is like to build a search engine around:

https://github.com/melling/SwiftResources/blob/master/swift_...

I'm using Swift to preprocess the data and I host my server on AppEngine using Go:

http://www.h4labs.com/dev/ios/swift.html

Since my engine is stitched together, I could use Python or Perl to scrape the sites and extract the words, ignoring JavaScript and css.

Re: Scrapy Tips from the Pros

#33

Post by stummjr » Wed, Jan 20, 2016, 2:08 PM UTC

kami8845 wrote: post #29

I love ScrapingHub (and use them) but these tips go completely against my own experience. Whenever I've tried to extract data like that inside Spiders I would invariably (and 50,000 URLs later) come to the realization that my .parse() ing code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. How to re-run all that with more robust lo…

What do you mean by "data like that"? Metadata in Microdata format?

Btw, nice to hear your own experience here. :)

Re: Scrapy Tips from the Pros

#34

Post by ddebernardy » Wed, Jan 20, 2016, 2:09 PM UTC

kami8845 wrote: post #29

I love ScrapingHub (and use them) but these tips go completely against my own experience. Whenever I've tried to extract data like that inside Spiders I would invariably (and 50,000 URLs later) come to the realization that my .parse() ing code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. How to re-run all that with more robust lo…

We actually do what you describe as well sometimes. In particular when we scrape sites with robust bot counter-measures to save on Crawlera [1] usage, or on crawls that take long enough that there's a genuine possibility that the site might change before you're done.

[1]: http://crawlera.com

Re: Scrapy Tips from the Pros

#35

Post by kami8845 » Wed, Jan 20, 2016, 2:13 PM UTC

stummjr wrote: post #33

kami8845 wrote: post #29

I love ScrapingHub (and use them) but these tips go completely against my own experience. Whenever I've tried to extract data like that inside Spiders I would invariably (and 50,000 URLs later) come to the realization that my .parse() ing code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. How to re-run all that with more robust lo…

What do you mean by "data like that"? Metadata in Microdata format? Btw, nice to hear your own experience here. :)

Microdata is indeed awesome but it's not always there. One website had location tags for me to extract from 97% of pages. Except for a few where the tags were just missing. Wound up using AlchemyAPI's entity extraction to try and get the location out of the text that way.

Thanks for the blogpost! I actually did not know about any of the 3 libraries and am gonna start using them.

Re: Scrapy Tips from the Pros

#36

Post by mikerice » Wed, Jan 20, 2016, 2:17 PM UTC

indymike wrote: post #30

Great to see Scrapy getting some love. It's really well done and it scales well (used it to scape ~2m job posts from ATS & government job banks in 2-3 hours).

Using it for the same use case, scraping a whole lot of job posts. Scrapy is love, scrapy is life.

Re: Scrapy Tips from the Pros

#37

Post by inovica » Wed, Jan 20, 2016, 2:17 PM UTC

Quick question. Could you use Scrapy to specific individual pages from thousands (or millions) of sites, or would you be better off using a search engine crawler like Nutch for this? I want to crawl the first page of a number of specific sites and was looking into the technologies for this.

Re: Scrapy Tips from the Pros

#38

Post by stummjr » Wed, Jan 20, 2016, 2:34 PM UTC

inovica wrote: post #37

Quick question. Could you use Scrapy to specific individual pages from thousands (or millions) of sites, or would you be better off using a search engine crawler like Nutch for this? I want to crawl the first page of a number of specific sites and was looking into the technologies for this.

Scrapy can do the job, for sure. We use it to crawl more than 2 billion pages a month. :)

Re: Scrapy Tips from the Pros

#39

Post by detaro » Wed, Jan 20, 2016, 2:41 PM UTC

pjc50 wrote: post #26

That reminds me, I was going to write a scraper to extract my HN comments.

HN API? https://github.com/HackerNews/API Haven't used it yet, so no comment on how well it works/how far back it goes)

Re: Scrapy Tips from the Pros

#40

Post by djm_ » Wed, Jan 20, 2016, 2:42 PM UTC

inovica wrote: post #37

Quick question. Could you use Scrapy to specific individual pages from thousands (or millions) of sites, or would you be better off using a search engine crawler like Nutch for this? I want to crawl the first page of a number of specific sites and was looking into the technologies for this.

Yes. Each spider in Scrapy has a "start_urls" parameter/method, so you'd just need to fill that up with all your domains and make sure the spider has freedom to crawl across domains. Each URL would be accessed, you'd do whatever you want to do and when the spider has visited them all, it would quit.