How to write a crawler • Hacker News Board

Re: How to write a crawler

#31

Post by EmanueleMinotto » Mon, Sep 30, 2013, 3:40 PM UTC

KMag wrote: post #30

asperous wrote: post #10

I wanna mention Nutch here: http://nutch.apache.org/ since it has been around for a while and a lot of thought was put into its design. For instance, people are discussing data stores, Nutch uses Hadoop. The web is probably bigger than you think, Google says "when our systems that process links on the web to find new content hit a milestone: 1 trillion (as in 1,000,000,000,000) unique URLs on the web at once!" (July,…

I used to be on the Google indexing team. Disregarding limits on the length of URLs, the size of the visible web is already infinite. For instance, there are many calendar pages out there that will happily give you month after month ad infinitum if you keep following the "next" link. Now, depending on how you prune your crawl to get rid of "uninteresting" content (such as infinite calendars) and how you deduplicate t…

You're right, I forgot to write it explicitly in the article but if someone will follow istructions (extract all tags and add them to the index) that method is tacit.

Re: How to write a crawler

#32

Post by peterwwillis » Mon, Sep 30, 2013, 5:49 PM UTC

Crawlers are one of those projects that's honestly best left to someone else. Fun for a hobby, but a nightmare to get right, and someone has already done the work for you. The exception is limited-use tools like Wget that can give you practical results for small-domain retrieval, but then kill you on CPU and memory and is impossible to scale; use a better tool or customize an existing one if you need to support large-scale crawls.

Some of the "little things" matter much more than your content analyzer or HTTP parsing - DNS performance and multi-homing being just a few that can have drastic effects.

Just as an example of how complex it gets, here's a brief overview of some of the features all crawlers should take into account: http://en.wikipedia.org/wiki/Web_crawler

Re: How to write a crawler

#33

Post by [deleted] » Mon, Sep 30, 2013, 11:42 PM UTC

EmanueleMinotto wrote: post #14

staunch wrote: post #11

The only complicated part that I've run into when writing crawlers is accidental tarpits. It's very easy to run into a situation in which you're repeatedly requesting the same content via many different URLs. For example, when a tracking parameter is added to any URL within the site: http://example.com/?cid=104484&pid=12002348&ref=1294902 http://example.com/?cid=104484&pid=12002348&ref=1294904 http://example.com/?cid…

If it's not included in robots.txt rules and doesn't have a canonical link that's not a problem, because the bot can't know if those pages are different or not so those pages you linked are different. This is the reason why crawlers can't try to fill forms. If you are really sure that these pages are the same, try checking the body content (if two or more pages have the same MD5 of the content, those pages are the sa…

[deleted]

Re: How to write a crawler

#34

Post by [deleted] » Mon, Sep 30, 2013, 11:43 PM UTC

TheLoneWolfling wrote: post #15

staunch wrote: post #11

The only complicated part that I've run into when writing crawlers is accidental tarpits. It's very easy to run into a situation in which you're repeatedly requesting the same content via many different URLs. For example, when a tracking parameter is added to any URL within the site: http://example.com/?cid=104484&pid=12002348&ref=1294902 http://example.com/?cid=104484&pid=12002348&ref=1294904 http://example.com/?cid…

Store a hash of each file you already have. If too many items from a single website collide, throw up a warning/error/flag for help/use some fuzzy method of identifying clashing URLs. If that's too slow/space-intensive, try a bloom filter.

[deleted]

Re: How to write a crawler

#35

Post by gondo » Mon, Sep 30, 2013, 11:43 PM UTC

cmiles74 wrote: post #26

gondo wrote: post #19

Earlier quoted context omitted.

why do you need Cassandra? isn't it possible to use different storage engines for ES? (disk + memory)

I was storing a lot of raw data (the content of the page crawled, PDF files) and there's no real gain to keeping it inside of the index.

hm im doing similar thing but im storing raw data in S3 and glacier. no need for nosql for this. or am i missing something?