How to get the top million sites list today? Alexa has shifted focus recently.
CommonCrawl / Open PageRank ?
Crawling a quarter billion webpages in 40 hours (2012)
51–60 of 68 posts
Re: Crawling a quarter billion webpages in 40 hours (2012)
#52I've been recently reading up on multithreading and multiprocessing in python. You mention that you've taken a multi threaded approach since the processes are i/o bound. Is this the same as running the script with asyncio as async/await?
Re: Crawling a quarter billion webpages in 40 hours (2012)
#53I've been recently reading up on multithreading and multiprocessing in python. You mention that you've taken a multi threaded approach since the processes are i/o bound. Is this the same as running the script with asyncio as async/await?
Oops, I meant to ask the author but realised that the author is not the same as the op. Hah!
Re: Crawling a quarter billion webpages in 40 hours (2012)
#54Earlier quoted context omitted.
I mean running bind[1] locally configured to act as a DNS cache. The operating system does some DNS caching as well, but it's not really tuned for crawling, and as a result it's very easy to end up spamming innocent DNS servers with an insane amount of lookup requests. [1] https://www.isc.org/bind/
ok but in my understanding isnt DNS also cached on the nearest wifi/ISP routers? the whole DNS system is just layer after layer of caches right? i.e. does caching on local machine actually matter? (real question, i dont know)
Yeah sure, but most of those caching layers (including possibly on the ISP level) aren't really configured for the DNS flood a crawling operation may result in.
If you're going to do way more DNS lookups than are expected from your connection, it's a good custom to provide your own caching layer that's scaled accordingly.
Not doing so probably won't break anything, but it risks degrading the DNS performance of other people using the same resolvers.
Re: Crawling a quarter billion webpages in 40 hours (2012)
#55Earlier quoted context omitted.
ok but in my understanding isnt DNS also cached on the nearest wifi/ISP routers? the whole DNS system is just layer after layer of caches right? i.e. does caching on local machine actually matter? (real question, i dont know)
Yeah sure, but most of those caching layers (including possibly on the ISP level) aren't really configured for the DNS flood a crawling operation may result in. If you're going to do way more DNS lookups than are expected from your connection, it's a good custom to provide your own caching layer that's scaled accordingly. Not doing so probably won't break anything, but it risks degrading the DNS performance of other…
gotcha. thanks for indulging my curiosity! hopefully others will learn good dns hygiene from this as well.
Re: Crawling a quarter billion webpages in 40 hours (2012)
#56A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results,…
>(even though the claim is that your IP is dynamic, in practice it hardly ever changes)
Every ISP just uses DHCP for router IPs. It's dynamic, you just have to let the lease time expire to renew it.
Or, have your own configurable router instead of the ISPs so that you can actually send a dhcp release command though they don't all support this part. Changing MAC Address will work otherwise.
Re: Crawling a quarter billion webpages in 40 hours (2012)
#57A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results,…
It makes very little difference what IP you scrape from, unless you're from a very dodgy subnet. The major content providers tend to go on a whitelist only based approach, you're either a human-like visitor or facing their anti-scraping methodologies.
I think the emphasis is on "never scrape from YOUR local/residential network".
Re: Crawling a quarter billion webpages in 40 hours (2012)
#58A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results,…
>(even though the claim is that your IP is dynamic, in practice it hardly ever changes) Every ISP just uses DHCP for router IPs. It's dynamic, you just have to let the lease time expire to renew it. Or, have your own configurable router instead of the ISPs so that you can actually send a dhcp release command though they don't all support this part. Changing MAC Address will work otherwise.
When the lease expires, the same IP is prioritized for renewal. Leases are generally for a week or two, but I've noticed dynamic IPs staying for 3 months or more. Swapping modems is really the best way to get a new external IP.
Re: Crawling a quarter billion webpages in 40 hours (2012)
#59A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results,…
I’ve learned a bunch of stuff about batch processing in the last few years that I would have sworn I already knew.
We had a periodic script that had all of these caveats about checking telemetry on the affected systems before running it, and even when it was happy it took gobs of hardware and ran for over 30 minutes.
There were all sorts of mistakes about traffic shaping that made it very bursty, like batching versus rate limiting, so the settings were determined by trial and error, essentially based on the 95th percentile of worst case (which is to say occasionally you’d get unlucky and knock things over). It also had to gather data from three services to feed the fourth and it was very spammy about that as well.
I reworked the whole thing with actual rate limiting, some different async blocks to interleave traffic to different services, and some composite rate limiting so we would call service C no faster than Service D could retire requests.
At one point I cut the cluster core count by 70% and the run time down to 8 minutes. Around a 12x speed up. Doing exactly the same amount of work, but doing it smarter.
CDNs and SaaS companies are in a weird spot where typical spider etiquette falls down. Good spiders limit themselves to N simultaneous requests per domain, trying to balance their burden across the entire internet. But they are capable of M*N total simultaneous requests, and so if you have a narrow domain or get unlucky they can spider twenty of your sites at the same time. Depending on how your cluster works (ie, cache expiry) that may actually cause more stress on the cluster than just blowing up one Host at a time.
People can get quite grumpy about this behind closed doors, and punishing the miscreants definitely gets discussed.
Re: Crawling a quarter billion webpages in 40 hours (2012)
#60I've been recently reading up on multithreading and multiprocessing in python. You mention that you've taken a multi threaded approach since the processes are i/o bound. Is this the same as running the script with asyncio as async/await?
At a 10,000 foot view (pedants will take offense) you should look to use multiprocessing for tasks which are CPU bound, and asyncio/threads (but really asyncio if you can) for problems which are IO-bound.
This is a massive simplification but most useful for a beginner.
Additionally, asyncio is not the same as multithreading, because typically asyncio is powered by a single-threaded event loop and use of a mechanism like select/kqueue/IOCP/epoll.