Viewing profile — shane42

This approach (scraped data on FS + metadata in DB) works well for storing scraped data. It was the first thing I prototyped when we started the project to move away from MongoDB. …

Take the example of crawl logs. Each log entry has a log level, timestamp and message. Typical use would be to view all ERROR (or higher) log levels, show all entries with a specif…

The problems with pagination are explained better on this SO post: http://stackoverflow.com/questions/7228169/slow-pagination-o... Mongo docs used the "walk" terminology

I would look at "delta-scraping" as one of the things that is easier with scrapy. There is an existing extension for it: https://github.com/scrapinghub/scrapylib/blob/master/scrapy…