Lessons Learned While Crawling the Web

#1

Post by jennita » Fri, Oct 04, 2013, 4:24 PM UTC

Lessons Learned While Crawling the Web
moz.com

Re: Lessons Learned While Crawling the Web

#2

Post by alecsmart1 » Fri, Oct 04, 2013, 6:18 PM UTC

The title is misleading. It should be "Lessons learned about web crawlers while crawling the web"

Re: Lessons Learned While Crawling the Web

#3

Post by jgmmo » Fri, Oct 04, 2013, 8:25 PM UTC

alecsmart1 wrote: post #2

The title is misleading. It should be "Lessons learned about web crawlers while crawling the web"

I think it should be called 'web scraping 101'

Very thin article.

Re: Lessons Learned While Crawling the Web

#4

Post by billrobertson42 » Sat, Oct 05, 2013, 5:54 PM UTC

jgmmo wrote: post #3

alecsmart1 wrote: post #2

The title is misleading. It should be "Lessons learned about web crawlers while crawling the web"

I think it should be called 'web scraping 101' Very thin article.

I found certain aspects of it really interesting. In particular, the way that the author interacts with the crawling software to gain insight into the structure of a website. Not just the process, but the way he interacts with it is interesting too.