Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to write a crawler

emanueleminotto.it

Post reply on HN
Print view View original

11–20 of 35 posts

Re: How to write a crawler

#11

Post by staunch »

The only complicated part that I've run into when writing crawlers is accidental tarpits. It's very easy to run into a situation in which you're repeatedly requesting the same content via many different URLs.

For example, when a tracking parameter is added to any URL within the site:

http://example.com/?cid=104484&pid=12002348&ref=1294902

http://example.com/?cid=104484&pid=12002348&ref=1294904

http://example.com/?cid=104484&pid=12002348&ref=1294905

http://example.com/?cid=104484&pid=12002348&ref=1294906

You can quickly get to billions of permutations for a single site. The canonical tag solves the problem when it's there, but I still haven't seen a simple solution to the problem when it's not.

Re: How to write a crawler

#12

Post by nostrademons »

EmanueleMinotto wrote: post #3
cmiles74 wrote: post #2

I respectfullly disagree. If ever there was a use-case for a NoSQL storage solution, web crawling certainly seems to be it. I've used Elasticsearch for indexing and Cassandra for storage, performance was more than good enough for our use-cases. It was easy to scale, as well.

A NoSQL solution is good because it's a DBMS (allows you to order collections). :) Filesystem is not good because you would need to order links' files in visited descendent order (not allowed in much filesystems), and to check if an URL is in the index you must store it with the MD5 as file name. A small DBMS like SQLite is not good for obvious reasons.

I would highly recommend storing the working set of links in RAM (with checkpointing to write it out to disk periodically). A Redis Set (for visited links) + Sorted Set (for unvisited links, ordered by priority) is perfect for this, since it lets you take up one full machine's RAM and does checkpointing automatically. If your crawl is too big to fit in RAM, get more machines and shard by URL hash. As others have pointed out, the file content itself should go in files, ideally ones that you can write to with straight appends.

The reason you don't want to hit the disk with each link (as both MySQL and PostGres usually do, barring caching) is that there can be hundreds to thousands of links on a page. A disk hit takes ~10ms; if you need to run hundreds of those, it's well over a second per page just to figure out which links on it are unvisited. Accessing main memory is about 100,000 times faster; even with sharding and RPC overhead for a distributed memory cache, you end up way ahead.

The reason to write the crawl text to an append-only log file is because disk seek times are bound by the rotation speed of the disk, which hasn't changed much recently, while disk bandwidth is bound by the rotation time of the disk divided by capacity, which has gone way up. So appends are much more efficient on disk than seeks are.

Re: How to write a crawler

#13

Post by at-fates-hands »

AcoonDe wrote: post #6

There are a few things that I hope you just left out in your description... - There is no mention of implementing a crawl-delay. You should always wait for several seconds (better yet, a minute) between requests to the same host. - Do you follow redirects when requesting the robots.txt? You should! Some sites send you a redirect to a different URL even for robots.txt. In most cases it is just a slightly different hos…

>>>You should always wait for several seconds (better yet, a minute) between requests to the same host.

This a thousand times.

I learned this lesson AFTER getting several angry emails from Admins and getting outright banned from one site for not having any delay between the requests in the first crawler I built.

Re: How to write a crawler

#14

Post by EmanueleMinotto »

staunch wrote: post #11

The only complicated part that I've run into when writing crawlers is accidental tarpits. It's very easy to run into a situation in which you're repeatedly requesting the same content via many different URLs. For example, when a tracking parameter is added to any URL within the site: http://example.com/?cid=104484&pid=12002348&ref=1294902 http://example.com/?cid=104484&pid=12002348&ref=1294904 http://example.com/?cid…

If it's not included in robots.txt rules and doesn't have a canonical link that's not a problem, because the bot can't know if those pages are different or not so those pages you linked are different. This is the reason why crawlers can't try to fill forms.

If you are really sure that these pages are the same, try checking the body content (if two or more pages have the same MD5 of the content, those pages are the same) or look for a form that generate those URLs.

Re: How to write a crawler

#15

Post by TheLoneWolfling »

staunch wrote: post #11

The only complicated part that I've run into when writing crawlers is accidental tarpits. It's very easy to run into a situation in which you're repeatedly requesting the same content via many different URLs. For example, when a tracking parameter is added to any URL within the site: http://example.com/?cid=104484&pid=12002348&ref=1294902 http://example.com/?cid=104484&pid=12002348&ref=1294904 http://example.com/?cid…

Store a hash of each file you already have. If too many items from a single website collide, throw up a warning/error/flag for help/use some fuzzy method of identifying clashing URLs.

If that's too slow/space-intensive, try a bloom filter.

Re: How to write a crawler

#16

Post by ebol4 »

AcoonDe wrote: post #6

There are a few things that I hope you just left out in your description... - There is no mention of implementing a crawl-delay. You should always wait for several seconds (better yet, a minute) between requests to the same host. - Do you follow redirects when requesting the robots.txt? You should! Some sites send you a redirect to a different URL even for robots.txt. In most cases it is just a slightly different hos…

>There are a LOT more file-extensions that you'll want to ignore.

It'd probably be a good idea just to whitelist filetypes that you actually want to crawl, rather than trying to blacklist all the ones you don't.

Re: How to write a crawler

#17

Post by drakaal »

AcoonDe wrote: post #6

There are a few things that I hope you just left out in your description... - There is no mention of implementing a crawl-delay. You should always wait for several seconds (better yet, a minute) between requests to the same host. - Do you follow redirects when requesting the robots.txt? You should! Some sites send you a redirect to a different URL even for robots.txt. In most cases it is just a slightly different hos…

For http://www.samuru.com we store our crawl in Google's DataStore. It gets big. By not storing all of the page, (we do content extraction first) we save quite a bit. Typically the content has much less markup than the template does.

We extract other things like the Opengraph image and social media links for author and store them independent of the content.

Re: How to write a crawler

#18

Post by boyter »

AcoonDe wrote: post #6

There are a few things that I hope you just left out in your description... - There is no mention of implementing a crawl-delay. You should always wait for several seconds (better yet, a minute) between requests to the same host. - Do you follow redirects when requesting the robots.txt? You should! Some sites send you a redirect to a different URL even for robots.txt. In most cases it is just a slightly different hos…

Its also missing some form of lock to prevent slow requests cascading (to the same host or IP) when the next instance of your crawler comes along. I did this once where I had a crawler spin up in a cron job. No problem, unless the previous crawler was still running.

Speaking from experience its far better to lock your crawler down from the beginning then to do it after you kill someones site. I did expose a few slow pages for some sites, but not before crashing the whole thing.

Re: How to write a crawler

#19

Post by gondo »

cmiles74 wrote: post #2

I respectfullly disagree. If ever there was a use-case for a NoSQL storage solution, web crawling certainly seems to be it. I've used Elasticsearch for indexing and Cassandra for storage, performance was more than good enough for our use-cases. It was easy to scale, as well.

why do you need Cassandra? isn't it possible to use different storage engines for ES? (disk + memory)

Re: How to write a crawler

#20

Post by gondo »

couple of additional points: - encoding - your input will have different encodings and its quite hard to guess the correct one, however you should at least try to convert everything into 1 encoding (f.e. utf8)

- by setting CURLOPT_ENCODING to '' you don't have to worry about (un)gzipping as curl ll do this for you (or it should)

- it might be a good idea to use url hash as url id (f.e. crc32)

- you should check content-length and content-type to avoid downloading huge files

btw your coding style is very disturbing. there shouldn't be spaces before or after ->

Post reply on HN