[deleted]
Crawling a quarter billion webpages in 40 hours (2012)
61–68 of 68 posts
Re: Crawling a quarter billion webpages in 40 hours (2012)
#62A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results,…
Most probably cloud based scraping services were not available in 2012. Now there are services available like scraperapi and others that don't need you to install anything at your end. You pay them, use their cloud infra, infinite proxies and even headless browsers. Shameless plug, I had written about it a few years ago on my blog post [1]
[1] https://blog.adnansiddiqi.me/scraping-dynamic-websites-using...
Re: Crawling a quarter billion webpages in 40 hours (2012)
#63To save you a click: Use 20 machines.
250M / 40hrs / 60min / 60s ~= 1,737 rps. That over 20 machines is ~87 rps per machine. Depending on a few factors, I rough out my backend Go stuff to handle between 1-5k rps per machine before we have real numbers.
You didn't write your Go stuff 11 years ago though...
Re: Crawling a quarter billion webpages in 40 hours (2012)
#64For all the people that say this is easy. Try it ! That's not easy at all, I've tried it and spend a few weeks to get similar performance. Receiving thousands of request is not similar to making thousands of requests, you can saturate your network, saturated with latency of random websites, get site that never timeout, parse multi megabytes malformed html, get infinite redirections.
My fastest implementation in python was actually using threads and was much faster than any async variant
Re: Crawling a quarter billion webpages in 40 hours (2012)
#65Earlier quoted context omitted.
250M / 40hrs / 60min / 60s ~= 1,737 rps. That over 20 machines is ~87 rps per machine. Depending on a few factors, I rough out my backend Go stuff to handle between 1-5k rps per machine before we have real numbers.
You didn't write your Go stuff 11 years ago though...
We started with Go version
1.2 which was released over ten years ago. Pretty darn close to 11 years.
Re: Crawling a quarter billion webpages in 40 hours (2012)
#66A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results,…
Not sure how is it in python, but what about using something like arti-client? Would it be already blocked?
Re: Crawling a quarter billion webpages in 40 hours (2012)
#67Re: Crawling a quarter billion webpages in 40 hours (2012)
#68Earlier quoted context omitted.
You didn't write your Go stuff 11 years ago though...
We started with Go version 1.2 which was released over ten years ago. Pretty darn close to 11 years. https://go.dev/doc/go1.2
Wow crazy how time flies. I did not realize Go has been in production for over 10 years now.