Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Crawling a quarter billion webpages in 40 hours (2012)

michaelnielsen.org

Post reply on HN
Print view View original

41–50 of 68 posts

Re: Crawling a quarter billion webpages in 40 hours (2012)

#41

Post by marginalia_nu »

swyx wrote: post #37
marginalia_nu wrote: post #14

Archive link: https://archive.is/yUWjh Also kinda wish the author paid any sort of attention to the fact that doing this incorrectly may create a flood of DNS queries. At least have the decency to set up a bind cache or something.

what is a bind cache? always assume most of us have terrible knowledge of networking

I mean running bind[1] locally configured to act as a DNS cache.

The operating system does some DNS caching as well, but it's not really tuned for crawling, and as a result it's very easy to end up spamming innocent DNS servers with an insane amount of lookup requests.

[1] https://www.isc.org/bind/

Re: Crawling a quarter billion webpages in 40 hours (2012)

#42

Post by ricardo81 »

m4r71n wrote: post #40

A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results,…

It makes very little difference what IP you scrape from, unless you're from a very dodgy subnet.

The major content providers tend to go on a whitelist only based approach, you're either a human-like visitor or facing their anti-scraping methodologies.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#44

Post by cacoos »

samwillis wrote: post #8

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…

I've always wondered why not to use request interceptors, get the html/json/xml/whatever url and then just call them.

If you need cookies/headers, you can always open the browser, log in and then make the same requests in the console, instead of waiting for the browser to load and scrape the UI (by xpath, etc.)

sounds weird going in circles: - SPA call some URL - SPA use the response data to populate the UI - You scrape the UI

instead of just calling the url inside the browser? am i missing something?

Re: Crawling a quarter billion webpages in 40 hours (2012)

#45

Post by marginalia_nu »

m4r71n wrote: post #40

A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results,…

I run a search engine crawler from my residential network. I get this too sometimes, but a lot of the time the IP shit-listing is temporary. It also seems to happen more often if you don't use a high enough crawl delay, ignore robots.txt, do deep crawls ignoring HTTP 429 errors and so on. You know, overall being a bad bot.

Overall, it's not as bad as it seems. I doubt anyone would accidentally damage their IP reputation doing otherwise above-board stuff.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#47

Post by zX41ZdbW »

luckystarr wrote: post #28

How to get the top million sites list today? Alexa has shifted focus recently.

Here is an example of how to obtain a list of the top six million domains from Tranco and analyze their content with ClickHouse: https://github.com/ClickHouse/ClickHouse/issues/18842

Re: Crawling a quarter billion webpages in 40 hours (2012)

#48

Post by sethammons »

onion2k wrote: post #7

To save you a click: Use 20 machines.

250M / 40hrs / 60min / 60s ~= 1,737 rps. That over 20 machines is ~87 rps per machine.

Depending on a few factors, I rough out my backend Go stuff to handle between 1-5k rps per machine before we have real numbers.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#50

Post by swyx »

marginalia_nu wrote: post #41
swyx wrote: post #37

Earlier quoted context omitted.

what is a bind cache? always assume most of us have terrible knowledge of networking

I mean running bind[1] locally configured to act as a DNS cache. The operating system does some DNS caching as well, but it's not really tuned for crawling, and as a result it's very easy to end up spamming innocent DNS servers with an insane amount of lookup requests. [1] https://www.isc.org/bind/

ok but in my understanding isnt DNS also cached on the nearest wifi/ISP routers? the whole DNS system is just layer after layer of caches right? i.e. does caching on local machine actually matter? (real question, i dont know)
Post reply on HN