How to crawl a quarter billion webpages in 40 hours (2012)
michaelnielsen.org
How to crawl a quarter billion webpages in 40 hours (2012)
1–10 of 25 posts
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#2This was written in 2012, Its even easier these days by using SQS and Cloud Formation. 250 Million is a small number you are better of first going through Common Crawl and then use data from crawls to build a better seed list.
Common Crawl now contains repeated crawls conducted every few months and also urls donated by blekko.
https://groups.google.com/forum/m/#!msg/common-crawl/zexccXg...
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#3This was written in 2012, Its even easier these days by using SQS and Cloud Formation. 250 Million is a small number you are better of first going through Common Crawl and then use data from crawls to build a better seed list. Common Crawl now contains repeated crawls conducted every few months and also urls donated by blekko. https://groups.google.com/forum/m/#!msg/common-crawl/zexccXg...
Do you have any recommendations on crawling / detecting changes on a large set of data? For example, the RSS feeds for every podcast on the planet?
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#4This was written in 2012, Its even easier these days by using SQS and Cloud Formation. 250 Million is a small number you are better of first going through Common Crawl and then use data from crawls to build a better seed list. Common Crawl now contains repeated crawls conducted every few months and also urls donated by blekko. https://groups.google.com/forum/m/#!msg/common-crawl/zexccXg...
Common Crawl contains the HTML?
I wonder how this is legal and considered acceptable. I wish I knew how even Google and others get away with scraping content, saving it, and utilizing it for profit without sharing any revenue with the original webmasters.
I know people can opt out of crawling, for those that actually respect that. But still, am I the only one who feels like this is wrong?
I guess I have this view that your domain is yours, and you invite the public in like an open house. It's my house, my property, and the door is open, where people can come in and look around at my stuff. But the expectation is that only locals will arrive, in small number, and they'll be good guests. If someone is breaking the lock on the bedroom door and going through the private drawers, that's wrong. If someone is taking photographs of everything, to then create a virtual tour of my house they charge for, that's wrong. The expectation is you're being nice by providing free and open access to information you created and own, and people should behave courteous to that.
Then if you as the webmaster choose, you can provide an API, or database dumps for people to download, along with the licensing terms. That is when it feels right for people to do things like this with the data, because you intentionally provided it through a non-personal interface.
To me the web is still a personal interface. I expect humans to use it, in an ordinary human-like way where it is somewhat ephemeral and courteous. I feel like Google cheated their way to success, and Common Crawl is stealing to rise their position in an unfair similar manner.
These all seem like parasites to me. They didn't create anything, they just steal it en masse.
There's so many businesses like that, such as Domain Tools that gets rich by hoarding everyone's contact details from WHOIS: http://whois.domaintools.com/commoncrawl.org
They have a screenshot history they won't ever delete even if you ask nicely. Here is a picture of Common Crawl from 2011: http://thumbnails.domaintools.com/domaintools/2016-01-08T19:...
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#5Web crawling is just like most things: 80% of the results for 20% of the work. It's always the last mile that takes the most significant cost and engineering effort.
as your index and scale grow you bump into the really difficult problems:
1. How do you handle so many DNS requests/sec without overloading upstream servers?
2. How do you discover and determine the quality of new links? It's only a matter of time until your crawler hits a treasure trove of spam domains.
3. How do you store, update, and access an index that's exponentially growing?
Just some ideas.
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#6Web crawling is just like most things: 80% of the results for 20% of the work. It's always the last mile that takes the most significant cost and engineering effort. as your index and scale grow you bump into the really difficult problems: 1. How do you handle so many DNS requests/sec without overloading upstream servers? 2. How do you discover and determine the quality of new links? It's only a matter of time until…
I've been working on something similar and I have ran into some of the issues you mention. As you correctly pointed out, quality and post processing is also relevant to not crawl irrelevant/spam sites, which can be HUGE! The work presented here is cool but it does not address the whole picture. Having a crawler that takes quality and user feedback into account is the hard part. Not to mention if you are being polite with the requests... we need to scale but not ignoring Robots.txt
So crawling a billion of links in X number of hours is not trivial but not that hard specially with cloud infrastructure like AWS, it's just a matter of a good enough implementation and how much money one wants to spend on it.
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#7[deleted]
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#8This was written in 2012, Its even easier these days by using SQS and Cloud Formation. 250 Million is a small number you are better of first going through Common Crawl and then use data from crawls to build a better seed list. Common Crawl now contains repeated crawls conducted every few months and also urls donated by blekko. https://groups.google.com/forum/m/#!msg/common-crawl/zexccXg...
IBM bought "Blekko" (semantic search engine) a few months ago to build a knowledge base (for Watson AI) and took them offline. (similar to Microsoft that bought "Powerset" for Bing and Cortana AI a few years ago)
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#9This was written in 2012, Its even easier these days by using SQS and Cloud Formation. 250 Million is a small number you are better of first going through Common Crawl and then use data from crawls to build a better seed list. Common Crawl now contains repeated crawls conducted every few months and also urls donated by blekko. https://groups.google.com/forum/m/#!msg/common-crawl/zexccXg...
Common Crawl contains the HTML? I wonder how this is legal and considered acceptable. I wish I knew how even Google and others get away with scraping content, saving it, and utilizing it for profit without sharing any revenue with the original webmasters. I know people can opt out of crawling, for those that actually respect that. But still, am I the only one who feels like this is wrong? I guess I have this view tha…
> I wonder how this is legal and considered acceptable. I wish I knew how Google and others gets away with scraping content [...]
Well, here's the answer: "transformative" reuse of content is explicitly permitted under copyright law. Simply reproducing the content and charging for it would not fall under this provision, but building an archive of publicly available information is - quite appropriately, permissible.
There was recently a very large court case regarding this principle and its application to Google Books. Google won, by demonstrating that their search index is not equivalent to and does not affect the market for the original work - a "transformative" use.
Sharing is good. Publicly available works achieve their aims only by being consumed by others - anyone who publishes a work free of charge should expect it to be, and remain, publicly accessible.
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#10This was written in 2012, Its even easier these days by using SQS and Cloud Formation. 250 Million is a small number you are better of first going through Common Crawl and then use data from crawls to build a better seed list. Common Crawl now contains repeated crawls conducted every few months and also urls donated by blekko. https://groups.google.com/forum/m/#!msg/common-crawl/zexccXg...
Common Crawl contains the HTML? I wonder how this is legal and considered acceptable. I wish I knew how even Google and others get away with scraping content, saving it, and utilizing it for profit without sharing any revenue with the original webmasters. I know people can opt out of crawling, for those that actually respect that. But still, am I the only one who feels like this is wrong? I guess I have this view tha…
Perhaps this is helpful? https://www.eff.org/deeplinks/2006/01/google-cache-ruled-fai...