Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrapy Tips from the Pros

blog.scrapinghub.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Scrapy Tips from the Pros

#41

Post by xrjn »

hokkos wrote: post #25

What I need is an API scraper, scrapy seems to be mostly for HTML. I know how to look at network requests in Chrome dev tools and JS function to understand the shape of REST API, so I need something to plan the exploration of the arguments space. For example if you want to scrap airbnb, you look at their API, find there is a REST call with a lat, long box, I need something to automatically explore an area, if the api…

Are you talking about something that would map out the API? The closest analogy to what I understood from your text was http://nodered.org/ in terms of design, I haven't seen anything like this for scraping yet. Definitely an interesting product to be made here if it doesn't exist yet.

Re: Scrapy Tips from the Pros

#42

Post by dante9999 »

kami8845 wrote: post #29

I love ScrapingHub (and use them) but these tips go completely against my own experience. Whenever I've tried to extract data like that inside Spiders I would invariably (and 50,000 URLs later) come to the realization that my .parse() ing code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. How to re-run all that with more robust lo…

> code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless.

Your data should not be worthless just because you dont catch some edge cases early. Sure there are always some edge cases but best way to handle them is to have proper validation logic in scrapy pipelines - if something is missing some required fields for example or you get some invalid values (e.g. prices as sequence of characters without digits) you should detect that immediately and not after 50k urls. Rule of thumb is: "never trust data from internet" and always validate it carefully.

If you have validation and encounter edge cases you will be sure that they are actual weird outliers that you can either choose to ignore or somehow try to force into your model of content.

Re: Scrapy Tips from the Pros

#43

Post by sheraz »

inovica wrote: post #37

Quick question. Could you use Scrapy to specific individual pages from thousands (or millions) of sites, or would you be better off using a search engine crawler like Nutch for this? I want to crawl the first page of a number of specific sites and was looking into the technologies for this.

Yes, if you use subclass CrawlSpider, then you will be able to set rules on your crawls [1]

[1] -http://scrapy.readthedocs.org/en/latest/topics/spiders.html?...

Re: Scrapy Tips from the Pros

#44

Post by blisterpeanuts »

Yay! Another article on scrapy. I'm just getting started and my first goal is to scrape a tedious web-based management console that I can't get API access to, and automate some tasks.

Very glad to learn about this site Scraping Hub. Keep the war stories coming. It's technologies like these that brighten up our otherwise drab tech careers and help some of us make it through the day.

Re: Scrapy Tips from the Pros

#45

Post by kami8845 »

dante9999 wrote: post #42
kami8845 wrote: post #29

I love ScrapingHub (and use them) but these tips go completely against my own experience. Whenever I've tried to extract data like that inside Spiders I would invariably (and 50,000 URLs later) come to the realization that my .parse() ing code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. How to re-run all that with more robust lo…

> code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. Your data should not be worthless just because you dont catch some edge cases early. Sure there are always some edge cases but best way to handle them is to have proper validation logic in scrapy pipelines - if something is missing some required fields for example or you get som…

Hmm, I'll have to investigate that, any tips for libraries to use for validation that tie well into scrapy?

What do you do if you discover that your parsing logic needs to be changed after you've scraped a few thousand items? Re-run your spiders on the URLs that raised errors?

Re: Scrapy Tips from the Pros

#46

Post by stummjr »

kami8845 wrote: post #45
dante9999 wrote: post #42

Earlier quoted context omitted.

> code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. Your data should not be worthless just because you dont catch some edge cases early. Sure there are always some edge cases but best way to handle them is to have proper validation logic in scrapy pipelines - if something is missing some required fields for example or you get som…

Hmm, I'll have to investigate that, any tips for libraries to use for validation that tie well into scrapy? What do you do if you discover that your parsing logic needs to be changed after you've scraped a few thousand items? Re-run your spiders on the URLs that raised errors?

Spider Contracts can help you: http://doc.scrapy.org/en/latest/topics/contracts.html

Re: Scrapy Tips from the Pros

#47

Post by JoshTriplett »

ddebernardy wrote: post #34
kami8845 wrote: post #29

I love ScrapingHub (and use them) but these tips go completely against my own experience. Whenever I've tried to extract data like that inside Spiders I would invariably (and 50,000 URLs later) come to the realization that my .parse() ing code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. How to re-run all that with more robust lo…

We actually do what you describe as well sometimes. In particular when we scrape sites with robust bot counter-measures to save on Crawlera [1] usage, or on crawls that take long enough that there's a genuine possibility that the site might change before you're done. [1]: http://crawlera.com

Crawling sites actively hostile to your crawler seems like it has both useful and shady applications. What kinds of things do you use that for?

Re: Scrapy Tips from the Pros

#49

Post by ddebernardy »

JoshTriplett wrote: post #47
ddebernardy wrote: post #34

Earlier quoted context omitted.

We actually do what you describe as well sometimes. In particular when we scrape sites with robust bot counter-measures to save on Crawlera [1] usage, or on crawls that take long enough that there's a genuine possibility that the site might change before you're done. [1]: http://crawlera.com

Crawling sites actively hostile to your crawler seems like it has both useful and shady applications. What kinds of things do you use that for?

On the one hand side there's no shortage of users who want to crawl popular sites to monitor e.g. search engine ranking or prices. Which is kind of shady in some sense, or not - when there's no API there's no other way...

On the other there are also areas of the web where crawlers are simply not welcome. For instance, DARPA uses a number of our technologies to monitor the dark web for criminal activities:

http://opencatalog.darpa.mil/MEMEX.html

Re: Scrapy Tips from the Pros

#50

Post by hackerboos »

steinsgate wrote: post #27

Confession : I am guilty of using regex superpowers to extract data from urls. Will check out w3lib soon!

I've had to write some gnarly XPath expressions to extract data with Scrapy.

> //b[contains(.,'City')]/following-sibling::a[not(preceding-sibling::b[contains(.,'Country')])]/text()

Post reply on HN