A Web Crawler with Asyncio Coroutines • Hacker News Board

Re: A Web Crawler with Asyncio Coroutines

#11

Post by fabiandesimone » Tue, Sep 15, 2015, 8:39 PM UTC

fsg7sdfg789 wrote: post #9

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

It's not a significant number of pages per day, honestly. For me, the limiting factor is almost always how many concurrent requests I feel comfortable making to the remote server. For big sites, the proxy I use generally caps it at 5 req / domain (concurrently). I generally use distributed crawlers, which means I can scale to millions of pages per day (assuming different domains). The biggest limiting factor is the d…

Thanks. Well, this does make me wonder if we are doing something wrong or we are performing actions that are slowing down the crawling. We have a good server (I believe).

Re: A Web Crawler with Asyncio Coroutines

#12

Post by kawera » Tue, Sep 15, 2015, 8:47 PM UTC

fabiandesimone wrote: post #11

fsg7sdfg789 wrote: post #9

Earlier quoted context omitted.

It's not a significant number of pages per day, honestly. For me, the limiting factor is almost always how many concurrent requests I feel comfortable making to the remote server. For big sites, the proxy I use generally caps it at 5 req / domain (concurrently). I generally use distributed crawlers, which means I can scale to millions of pages per day (assuming different domains). The biggest limiting factor is the d…

Thanks. Well, this does make me wonder if we are doing something wrong or we are performing actions that are slowing down the crawling. We have a good server (I believe).

Don't forget to check your connection too; maybe you are filling it up or have latency issues.

Re: A Web Crawler with Asyncio Coroutines

#13

Post by troels » Tue, Sep 15, 2015, 8:50 PM UTC

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

I have a crawler setup the pulls a few million pages per day. The main constraint is not in the crawler setup, but rather in how much load the subject sites can withstand. If I don't throttle down the traffic, the sites will be dos'ed very quickly. Of course, this is mainly a problem because I crawl a lot of pages from each site - if you have a crawler that crawls a few pages from a lot of sites, you would have a different scenario.

Re: A Web Crawler with Asyncio Coroutines

#14

Post by troels » Tue, Sep 15, 2015, 8:55 PM UTC

fabiandesimone wrote: post #11

fsg7sdfg789 wrote: post #9

Earlier quoted context omitted.

It's not a significant number of pages per day, honestly. For me, the limiting factor is almost always how many concurrent requests I feel comfortable making to the remote server. For big sites, the proxy I use generally caps it at 5 req / domain (concurrently). I generally use distributed crawlers, which means I can scale to millions of pages per day (assuming different domains). The biggest limiting factor is the d…

Thanks. Well, this does make me wonder if we are doing something wrong or we are performing actions that are slowing down the crawling. We have a good server (I believe).

You might want to benchmark where your software is spending its time. A typical overhead is in the connection time. You may be able to speed things up with a local dns cache and by using http keep-alive. You also generally want to do a lot of parallel requests, since most time would be spent waiting for the subject site to respond.

Re: A Web Crawler with Asyncio Coroutines

#15

Post by xitep » Tue, Sep 15, 2015, 8:56 PM UTC

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

The article is a very nice tutorial on asynch-io. I really enjoyed it. But as the authors themselves said the web-crawler they built in this article is just a toy.

Writing an efficient and _well behaved_ web crawler is imho quite a complicated undertaking. Others here have already pointed out that it's more or less a scalability problem, hence, a number doesn't make sense. reinhardt has provided a list of links - which from a quick glance - look very interesting and might bring you further.

Re: A Web Crawler with Asyncio Coroutines

#16

Post by fabiandesimone » Tue, Sep 15, 2015, 9:10 PM UTC

reinhardt wrote: post #8

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

It doesn't make much sense to give a number for speed without some specifics about the crawler environment, such as: - How many servers (if distributed)? - How many cores/server? - What kind of processing takes place for each page? Does it just download and save the pages somewhere (local filesystem, cloud storage, database) or it extracts (semi) structured data? And so on. Specifics aside, these days it's not hard t…

Thank you very much!

Re: A Web Crawler with Asyncio Coroutines

#17

Post by fsg7sdfg789 » Tue, Sep 15, 2015, 10:25 PM UTC

fabiandesimone wrote: post #11

fsg7sdfg789 wrote: post #9

Earlier quoted context omitted.

It's not a significant number of pages per day, honestly. For me, the limiting factor is almost always how many concurrent requests I feel comfortable making to the remote server. For big sites, the proxy I use generally caps it at 5 req / domain (concurrently). I generally use distributed crawlers, which means I can scale to millions of pages per day (assuming different domains). The biggest limiting factor is the d…

Thanks. Well, this does make me wonder if we are doing something wrong or we are performing actions that are slowing down the crawling. We have a good server (I believe).

I just reached out to you via email to see if I can help.

Re: A Web Crawler with Asyncio Coroutines

#18

Post by jdrock » Tue, Sep 15, 2015, 10:45 PM UTC

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

Here's how many URLs we crawl every second with 80legs (http://www.80legs.com)

https://dl.dropboxusercontent.com/u/44889964/Descartes%20%20...

This translates to about 700MM/month. The bump you see this month is just us adding more crawling nodes to our cluster.

Re: A Web Crawler with Asyncio Coroutines

#19

Post by mseri » Tue, Sep 15, 2015, 10:55 PM UTC

Animats wrote: post #4

It would be interesting to compare this Python approach with a Go goroutine approach. The main question is whether Go's libraries handle massive numbers of connections well. Since Google wrote Go to be used internally, they probably do.

Or Erlang, or rust

Re: A Web Crawler with Asyncio Coroutines

#20

Post by chrismarlow9 » Tue, Sep 15, 2015, 11:10 PM UTC

fabiandesimone wrote: post #5

I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…

Assuming you're not bound by rate limiting on the remote hosts and the average page crawled is I've written more web crawlers than I can count in php, python, scala, golang, nodejs, and perl. Right now assuming you want to just gather some form of JSON/HTML from the response, I would use golang and gokogiri with XPaths (and of course json unmarshal for json). It will make you laugh at 120k per day. Feel free to ping me if you would like to discuss making me one of those freelancers.