A Web Crawler with Asyncio Coroutines
aosabook.org
A Web Crawler with Asyncio Coroutines
1–10 of 31 posts
Re: A Web Crawler with Asyncio Coroutines
#2Great tutorial, I would love to see this rewritten with the new async / await syntax in python 3.5
Re: A Web Crawler with Asyncio Coroutines
#3Appreciate the in-depth description. Look forward to working through this in detail.
Re: A Web Crawler with Asyncio Coroutines
#4It would be interesting to compare this Python approach with a Go goroutine approach. The main question is whether Go's libraries handle massive numbers of connections well. Since Google wrote Go to be used internally, they probably do.
Re: A Web Crawler with Asyncio Coroutines
#5I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers).
While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling.
My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day?
The crawler I built is doing about 120K pages per day which to our initial needs is not bad at all, but wonder if in the crawling world this is peanuts or a decent chunk of pages?
Re: A Web Crawler with Asyncio Coroutines
#6I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…
I have scrapers built in Python that do well over a million pages per day, but that's not really a benchmark you can use. It all depends on the amount of computation required to extract the page data among other things.
You should be able to achieve > 120k per day for sure though. That's less than two per second.
Re: A Web Crawler with Asyncio Coroutines
#7I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…
I have scrapers built in Python that do well over a million pages per day, but that's not really a benchmark you can use. It all depends on the amount of computation required to extract the page data among other things. You should be able to achieve > 120k per day for sure though. That's less than two per second.
Thank you. Well, I'm doing several things:
1) I check whether or not the page we just scrapped has any of the tags we are looking for.
2) We then extract any information within those tags (images, etc.)
3) We follow trough every link and if it's not in the seen/scrapped list, we add them to the queue.
Not sure if this helps to narrow it down.
Thanks!
Re: A Web Crawler with Asyncio Coroutines
#8I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…
It doesn't make much sense to give a number for speed without some specifics about the crawler environment, such as:
- How many servers (if distributed)?
- How many cores/server?
- What kind of processing takes place for each page?
Does it just download and save the pages somewhere (local filesystem, cloud storage, database) or it extracts (semi) structured data? And so on.
http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-bil...
http://blog.semantics3.com/how-we-built-our-almost-distribut...
http://engineering.bloomreach.com/crawling-billions-of-pages...
http://engineering.bloomreach.com/crawling-billions-of-pages...
Re: A Web Crawler with Asyncio Coroutines
#9I'm working on a project that involves lot's of web crawling. I'm not technical at all (I'm hiring freelancers). While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling. My question is: in terms of crawling speed (and I know this is dependent of several factors) what's a decent amount of pages a good crawler could do per day? The crawler I built is…
It's not a significant number of pages per day, honestly. For me, the limiting factor is almost always how many concurrent requests I feel comfortable making to the remote server. For big sites, the proxy I use generally caps it at 5 req / domain (concurrently).
I generally use distributed crawlers, which means I can scale to millions of pages per day (assuming different domains). The biggest limiting factor is the database layer, how many writes can I do in a day.
If I need to go faster, I just spin up another crawler worker, which connects to the queue and starts pulling jobs.
I believe anything under a million pages / day should be do-able by a homebuilt, single-server system.
Re: A Web Crawler with Asyncio Coroutines
#10Node is well-suited for this type of thing and there are numerous libraries to help.