A Web Crawler with Asyncio Coroutines • Hacker News Board

Re: A Web Crawler with Asyncio Coroutines

#21

Post by fabiandesimone » Tue, Sep 15, 2015, 11:24 PM UTC

chrismarlow9 wrote: post #20

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

Assuming you're not bound by rate limiting on the remote hosts and the average page crawled is I've written more web crawlers than I can count in php, python, scala, golang, nodejs, and perl. Right now assuming you want to just gather some form of JSON/HTML from the response, I would use golang and gokogiri with XPaths (and of course json unmarshal for json). It will make you laugh at 120k per day. Feel free to ping…

I don't see any way to ping you. Do you have an email?

Re: A Web Crawler with Asyncio Coroutines

#22

Post by fabiandesimone » Tue, Sep 15, 2015, 11:25 PM UTC

jdrock wrote: post #18

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

Here's how many URLs we crawl every second with 80legs ( http://www.80legs.com ) https://dl.dropboxusercontent.com/u/44889964/Descartes%20%20... This translates to about 700MM/month. The bump you see this month is just us adding more crawling nodes to our cluster.

Looks pretty great!

Re: A Web Crawler with Asyncio Coroutines

#23

Post by logn » Wed, Sep 16, 2015, 12:20 AM UTC

mseri wrote: post #19

Animats wrote: post #4

It would be interesting to compare this Python approach with a Go goroutine approach. The main question is whether Go's libraries handle massive numbers of connections well. Since Google wrote Go to be used internally, they probably do.

Or Erlang, or rust

Or Java. Right now I have a web driver that uses standard Java classes for requests but I wonder if NIO would offer significantly better performance.

Re: A Web Crawler with Asyncio Coroutines

#24

Post by logn » Wed, Sep 16, 2015, 12:31 AM UTC

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

It all depends how many servers you're using (or how much memory/CPU each has), whether it's architected properly for horizontal scaling, the performance of your proxy servers (if applicable), how much you're stressing the target website, how efficient your HTML parsing is, and whether you need to render CSS/JS pages.

I'm just now finishing a project for an ISP building a cache of webpages using my project jBrowserDriver. They can basically turn on as many VMs as they need to horizontally scale out, and the servers all seamlessly load balance themselves and pull work off a central queue. One important part is to handle failures and crashes, isolating impact to everything else. In this approach, separate OS processes are helpful.

Re: A Web Crawler with Asyncio Coroutines

#25

Post by Schwolop » Wed, Sep 16, 2015, 4:37 AM UTC

This article is way more important than the web crawler example used to motivate it. It's easily the single best thing I've ever read on asyncio, and I've been using it in anger for a year now. I've passed it around my team, and will be recommending it far and wide!

Re: A Web Crawler with Asyncio Coroutines

#26

Post by atombender » Wed, Sep 16, 2015, 4:41 AM UTC

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

One tip: If you make your pipeline fine-grained, you have much more flexibility in terms of scheduling and parallelization, and also makes it easier to expand and design.

By fine-grained I mean that fetching, crawling, extraction and whatever other processing you're doing should be separate, discrete steps.

Example naive topology:

Fetcher: Pops next URL off a queue, fetches it, stores the raw data somewhere, emits a "fetched" event.

Link extractor: Subscribes to fetch events, extracts every URL from the data, each of which is emitted as a "link" event.

Crawling scheduler: Listens to link events, schedules "fetch" events for each URL. This is where you might add filtering and prioritization rules, for example.

Now you have three queues and three consumers, which can run in parallel with any number of worker processes dedicated to them. A naive solution could use something like a database for the events, but a dedicated queue such as RabbitMQ would fare better.

Re: A Web Crawler with Asyncio Coroutines

#27

Post by greglindahl » Wed, Sep 16, 2015, 5:29 AM UTC

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

Academics who write crawlers that don't do much with the pages they fetch can do 100s of millions of pages in a day with an ordinary server and a big, fat network pipe. At that speed they aren't even parsing html, they're using regexes to try to find URLs and that's about it.

At blekko, we did ~ 100k pages/day/server with our production crawler, running on a cluster which was also doing anti-web-spam, inverting outgoing links into incoming links, indexing everything, and analytics batch jobs supporting development.

So unless you're doing a LOT of work on every webpage, you're kinda slow.

The easiest mistake to make is to not be asynch enough. This Python example is great.

Re: A Web Crawler with Asyncio Coroutines

#28

Post by adamseabrook » Wed, Sep 16, 2015, 6:03 AM UTC

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

meanpath.com can do around 200 million pages per day using 13 fairly average dedicated servers. We only crawl the front page (mile wide, inch deep) so the limiting factor is actually DNS. Looking at the network traffic the bandwidth is split evenly between DNS and HTTP. Google public DNS will quickly rate limit you so you need to use your own resolvers (we use Unbound).

Unlike Blekko we are just capturing the source and dumping it into a DB without doing any analysis. As soon as you start trying to parse anything in the crawl data your hardware requirements go through the roof. parallel with wget or curl is enough to crawl millions of pages per day. I often use http://puf.sourceforge.net/ when I need to do a quick crawl

"puf -nR -Tc 5 -Tl 5 -Td 20 -t 1 -lc 200 -dc 5 -i listofthingstodownload" will easily do 10-20 million pages per day if you are spreading your requests across a lot of hosts.

Re: A Web Crawler with Asyncio Coroutines

#29

Post by greglindahl » Wed, Sep 16, 2015, 6:51 AM UTC

adamseabrook wrote: post #28

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

meanpath.com can do around 200 million pages per day using 13 fairly average dedicated servers. We only crawl the front page (mile wide, inch deep) so the limiting factor is actually DNS. Looking at the network traffic the bandwidth is split evenly between DNS and HTTP. Google public DNS will quickly rate limit you so you need to use your own resolvers (we use Unbound). Unlike Blekko we are just capturing the source…

We used djbdns on every crawl machine, and did not find DNS to be limiting at all. You should also make sure there isn't any connection tracking, or firewalls/middleboxes which are doing connection-based anything, or NAT, or really anything other than raw Internet between you and the Internet.

Re: A Web Crawler with Asyncio Coroutines

#30

Post by potatosareok » Wed, Sep 16, 2015, 7:36 AM UTC

One question I have about this - and I might have missed in article is - I'm all for using asyncio to make HTTP requests. But I see they apparently also use asyncio for "parse_links". Since parselinks should be CPU op, would it make sense to use fibers to download links and pass them into a thread pool to actually parse them//add to queue?

I'm messing around with some of the ParallelUniverse Java fiber implementation and what I do is spam fibers to download pages and send the String response over to another fiber over a channel that maintains a thread pool to parse response body as they come in//create new fibers to read these links.

I'm really just doing this to get more familiar with async programming and specifically the paralleluniverse Java libs but one thing I'm struggling a bit with is how to best make it well behaved (e.g right now there's no bound on number of outstanding HTPT requests).