How to write a crawler
emanueleminotto.it
How to write a crawler
1–10 of 35 posts
Re: How to write a crawler
#2I respectfullly disagree. If ever there was a use-case for a NoSQL storage solution, web crawling certainly seems to be it. I've used Elasticsearch for indexing and Cassandra for storage, performance was more than good enough for our use-cases. It was easy to scale, as well.
Re: How to write a crawler
#3I respectfullly disagree. If ever there was a use-case for a NoSQL storage solution, web crawling certainly seems to be it. I've used Elasticsearch for indexing and Cassandra for storage, performance was more than good enough for our use-cases. It was easy to scale, as well.
A NoSQL solution is good because it's a DBMS (allows you to order collections). :)
Filesystem is not good because you would need to order links' files in visited descendent order (not allowed in much filesystems), and to check if an URL is in the index you must store it with the MD5 as file name.
A small DBMS like SQLite is not good for obvious reasons.
Re: How to write a crawler
#4Udacity CS101 [1] also goes through the basics of building a web crawler. It's a lot more lightweight (no backend, etc), but it's a fun overview and can be completed pretty quickly.
Re: How to write a crawler
#5You definitely want to store raw text in flat files and store metadata in a database. By metadata I'm not referring to only the values found in the html page's meta tags but other things like page hash, word count, link count, etc. It all depends on what you are doing. If it's in a database you will have a very very big file for that archives table (mongodb and mysql+innodb come to mind).
Re: How to write a crawler
#6There are a few things that I hope you just left out in your description...
- There is no mention of implementing a crawl-delay. You should always wait for several seconds (better yet, a minute) between requests to the same host.
- Do you follow redirects when requesting the robots.txt? You should! Some sites send you a redirect to a different URL even for robots.txt. In most cases it is just a slightly different hostname, like www.domain.com instead of domain.com. But it can redirect you to somewhere completely different in some cases.
- You probably don't want to crawl anything that ends with .jpg, .gif, and definitely not something like .avi, .wmv or .mkv. There are a LOT more file-extensions that you'll want to ignore.
I agree with cmiles74 that using a database is probably a bad idea. For a sizeable crawl (say a billion pages) this database will get pretty damn big. I doubt that you will be able to get decent performance out of anything with "SQL" in its name for such a use-case, unless you throw a ton of hardware at it. Building your own specialized solution for this would probably be a lot faster and less resource-intensive.
Re: How to write a crawler
#7[deleted]
Re: How to write a crawler
#8You definitely want to store raw text in flat files and store metadata in a database. By metadata I'm not referring to only the values found in the html page's meta tags but other things like page hash, word count, link count, etc. It all depends on what you are doing. If it's in a database you will have a very very big file for that archives table (mongodb and mysql+innodb come to mind).
in a couple of projects I worked on, we also stored visited urls in a set of bloom filters, also stored in flat files on disk.
At some point querying the db to check what URLs you have can become quite heavy
Re: How to write a crawler
#9The Content Extractor is overly simple. You don't want all the Anchor tags for most things, you want the ones that aren't part of the page template.
It took a lot of man hours to build the content extractor we use for our search engine.
https://www.mashape.com/stremor/stremor-content-extractor
Also because not every site has a SiteMap, or well linked site structure you may have to turn to Social like FB and Twitter if you want to get everything.
Re: How to write a crawler
#10I wanna mention Nutch here: http://nutch.apache.org/
since it has been around for a while and a lot of thought was put into its design. For instance, people are discussing data stores, Nutch uses Hadoop.
The web is probably bigger than you think, Google says "when our systems that process links on the web to find new content hit a milestone: 1 trillion (as in 1,000,000,000,000) unique URLs on the web at once!" (July, 2008)
You might consider just crawling certain parts of the web, or using a search engine (api, like Yahoo! BOSS) to gather relevant links and crawl from there, using a depth limit. Just an idea.