What kinds of careers often deal with web scraping and doing these sorts of tasks? I am really interested in the field and some of you seem to be real experts in this field.
Scrapy Tips from the Pros
71–78 of 78 posts
Re: Scrapy Tips from the Pros
#72I like this article but for its discussion of these libraries. On another note... Am I the only one who dislikes Scrapy? I think it's basically the iOS of scraping tools: It's incredibly easy to setup and use, and then as soon as you need to do something even minutely non-standard it reveals itself to be frustratingly inflexible.
Scrapy is about as flexible and extensible as you can get... Care to elaborate on "frustratingly inflexible"?
Image pipeline converts to jpg automatically. Cannot be disabled without writing all the image code yourself
Re: Scrapy Tips from the Pros
#73I like this article but for its discussion of these libraries. On another note... Am I the only one who dislikes Scrapy? I think it's basically the iOS of scraping tools: It's incredibly easy to setup and use, and then as soon as you need to do something even minutely non-standard it reveals itself to be frustratingly inflexible.
I don't really know what you mean. Scrapy is in Python. You can do whatever you want.
Re: Scrapy Tips from the Pros
#74Earlier quoted context omitted.
Much like apples to oranges. Nokogiri is a tag soup parser. Scrapy is a web scraping framework. In addition to tag soup parsing, Scrapy handles a slew of things such as text encoding problems, retrying urls that fail owing to network problems (if you wish), dispatching requests across multiple spiders with a shared crawl frontier (see Frontera), shared code between similar spiders using middlewares and pipelines, and…
Makes sense, thanks for the clarification. To your knowledge, is there anything comparable to Scrapy in Ruby land?
Not that we're aware of. Most rubyists use request and a tag soup parser, without benefiting from any type of parallelization that you get from Scrapy.
Re: Scrapy Tips from the Pros
#75What kinds of careers often deal with web scraping and doing these sorts of tasks? I am really interested in the field and some of you seem to be real experts in this field.
All sorts of careers, for example:
- developers who want to develop some data-based product (a travel agency website, who finds the best deals from airline companies);
- lawyers can use it to structure the data from Judgments and Laws (so that they are able to query the data for things like: which judges have interpreted this law in their judgments) (more on this: http://blog.scrapinghub.com/2016/01/13/vizlegal-rise-of-mach...)
- (data-)journalists who work on investigative data-based articles (they use it to gather the data to build visualizations, infographics, and also to support their arguments).
- real state agencies can use it to grab the prices of their competitors, or to get a map of what people are selling, what are the areas where there is more demand.
- large companies that want to track their online reputation can scrape forums, blogs, etc, for further analysis.
- online retailers that want to keep their prices balanced with their competitors can scrape the competitors websites collecting prices from them.
More on Quora: https://www.quora.com/What-are-examples-of-how-real-business...
Re: Scrapy Tips from the Pros
#76That reminds me, I was going to write a scraper to extract my HN comments.
Search API https://hn.algolia.com/api
BigQuery https://news.ycombinator.com/item?id=10440502 (2015-Oct)
Re: Scrapy Tips from the Pros
#77Earlier quoted context omitted.
Great article, thanks for sharing. What would you recommend for people who need to operate a cluster of scrapyd instances? Are there any recommended ways of managing the distribution of tasks to multiple scrapyd instances? I have looked at scrapyd-cluster, but I would prefer not to add a Zookeeper cluster to my stack. Currently I'm thinking of modifying scrapyd-cluster so that it uses AMQP (via Celery) to handle task…
Hi, mryan! I'm the core Frontera developer. The precise answer heavily depends on your use case (what "task" is? scalability requirements, data flow), so please ask your question in Frontera google groups, and we will try to address it directly. First you could try making use of Frontera, here are the different distribution models it provides out of the box http://frontera.readthedocs.org/en/latest/topics/run-modes.h…
Sounds like I have some more research to do - thanks for the detailed response.
Btw, you were right, I meant scrapy-cluster and not scrapyd-cluster: https://github.com/istresearch/scrapy-cluster. There is a requirement on Zookeeper/Kafka, which is the main showstopper for me.
Re: Scrapy Tips from the Pros
#78I like this article but for its discussion of these libraries. On another note... Am I the only one who dislikes Scrapy? I think it's basically the iOS of scraping tools: It's incredibly easy to setup and use, and then as soon as you need to do something even minutely non-standard it reveals itself to be frustratingly inflexible.
Scrapy is about as flexible and extensible as you can get... Care to elaborate on "frustratingly inflexible"?
I do a lot of scraping specific pages and often have to auth, form-fill, refresh, recurse, use a custom SSL/TLS adapter, etc., in order to get what I'm after. I'm sure Scrapy would be great if I just had a giant queue of GET requests. Also, don't get me started on the Reactor.