21–30 of 35 posts
Udacity CS101 [1] also goes through the basics of building a web crawler. It's a lot more lightweight (no backend, etc), but it's a fun overview and can be completed pretty quickly. [1]: https://www.udacity.com/course/cs101
Multi-threaded. Built-in page delay (200ms default). Does HTTPS, headers, POST, cookies, follows robots.txt .
I prefer it to nutch for small to medium sized jobs.
> id is an incremental value, I choose 11 as a length
for this primary key but this value is defined
by the number of pages you’ll need to index
Also the visited field would be better to be represented by a timestamp, choosing the right datatype does matter in large tables.
I wanna mention Nutch here: http://nutch.apache.org/ since it has been around for a while and a lot of thought was put into its design. For instance, people are discussing data stores, Nutch uses Hadoop. The web is probably bigger than you think, Google says "when our systems that process links on the web to find new content hit a milestone: 1 trillion (as in 1,000,000,000,000) unique URLs on the web at once!" (July,…
Any numbers?
I respectfullly disagree. If ever there was a use-case for a NoSQL storage solution, web crawling certainly seems to be it. I've used Elasticsearch for indexing and Cassandra for storage, performance was more than good enough for our use-cases. It was easy to scale, as well.
why do you need Cassandra? isn't it possible to use different storage engines for ES? (disk + memory)
You definitely want to store raw text in flat files and store metadata in a database. By metadata I'm not referring to only the values found in the html page's meta tags but other things like page hash, word count, link count, etc. It all depends on what you are doing. If it's in a database you will have a very very big file for that archives table (mongodb and mysql+innodb come to mind).
in a couple of projects I worked on, we also stored visited urls in a set of bloom filters, also stored in flat files on disk. At some point querying the db to check what URLs you have can become quite heavy
You definitely want to store raw text in flat files and store metadata in a database. By metadata I'm not referring to only the values found in the html page's meta tags but other things like page hash, word count, link count, etc. It all depends on what you are doing. If it's in a database you will have a very very big file for that archives table (mongodb and mysql+innodb come to mind).
Most of them use the same data-structures (e.g. b+trees) and especially the newer ones turn into copy-on-write systems like e.g. CouchDB
I guess if you want to have backups of that table, you probably would like to keep it at a manageable size, but why not just save the html blobs in a separate table?
The only complicated part that I've run into when writing crawlers is accidental tarpits. It's very easy to run into a situation in which you're repeatedly requesting the same content via many different URLs. For example, when a tracking parameter is added to any URL within the site: http://example.com/?cid=104484&pid=12002348&ref=1294902 http://example.com/?cid=104484&pid=12002348&ref=1294904 http://example.com/?cid…
If it's not included in robots.txt rules and doesn't have a canonical link that's not a problem, because the bot can't know if those pages are different or not so those pages you linked are different. This is the reason why crawlers can't try to fill forms. If you are really sure that these pages are the same, try checking the body content (if two or more pages have the same MD5 of the content, those pages are the sa…
I wanna mention Nutch here: http://nutch.apache.org/ since it has been around for a while and a lot of thought was put into its design. For instance, people are discussing data stores, Nutch uses Hadoop. The web is probably bigger than you think, Google says "when our systems that process links on the web to find new content hit a milestone: 1 trillion (as in 1,000,000,000,000) unique URLs on the web at once!" (July,…
Now, depending on how you prune your crawl to get rid of "uninteresting" content (such as infinite calendars) and how you deduplicate the pages you find, you'll come up with vastly varying estimates of how big the visible web is.
Edit: on a side note, don't crawl the web using a naive depth-first search. You'll get stuck in some uninteresting infinitely deep branch of the web.