Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Crawling a quarter billion webpages in 40 hours (2012)

michaelnielsen.org

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Crawling a quarter billion webpages in 40 hours (2012)

#21

Post by marginalia_nu »

abhibeckert wrote: post #18
marginalia_nu wrote: post #13

Earlier quoted context omitted.

Depends on what you want with the data I guess. From a search engine point of view, the SPA case isn't that relevant, since SPAs can't reliably be linked to and in general tend to not be very stable and it's overall difficult to figure out how to enumerate and traverse their views. I think a good middle ground might be to do a first pass with a "stupid" crawler, and then re-visit the sites where you were blocked or t…

The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content. If Google can crawl it, then you can too. And while Google doesn't use a headless browser (or at least I assume they don't) they absolutely do execute javascript before loading the conte…

> The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content.

This is true for some SPAs, but not all SPAs, and there's not really any way of telling which is which.

I don't personally attempt to crawl SPAs because it's not the sort of content I want to index.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#23

Post by jaymzcampbell »

samwillis wrote: post #8

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…

On the topic of JSON and whatnot that reminded me of this awesome post[1] about looking within heap snapshots for entire data structures where they may have a lot of nicely structured data within it not readily available from more direct URL calls.

[1] https://www.adriancooney.ie/blog/web-scraping-via-javascript...

(Previous comments about it from 2022 here: https://news.ycombinator.com/item?id=31205139)

Re: Crawling a quarter billion webpages in 40 hours (2012)

#24

Post by arek_nawo »

samwillis wrote: post #8

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…

Which way to go will depend on your use-case and what websites you want to scrape.

For occasional web crawling, headless browser is great as it's easy to set up and you're almost guaranteed to get the data you need.

For frequent or large-scale crawling, it's a different story. Surely you can implement something with just HTTP library, but you'll need to test it throughly and make some research before hand. That said, most scraped, content-heavy websites use either static HTML or SSR, in which case you can use HTTP no problem.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#25

Post by sethammons »

marginalia_nu wrote: post #19
ddorian43 wrote: post #16

With async in any language of choice this should need just 1 server. 250M/40hours=1.7K requests/second. You can probably do it on a single core in a low level language like Rust/Java.

Threads isn't the only bottleneck in crawling. Assuming you're crawling at a civilized rate of about 1 request/second, you're only so many network hiccups away from consuming the entire ephemeral port range with connections in TIME_WAIT or CLOSE_WAIT.

Crank up the ulimit. And what is this 1 req/second nonsense? 1 req/sec/domain maybe. I have to agree, my first thought is "why this is not a single node in Go?"

Re: Crawling a quarter billion webpages in 40 hours (2012)

#26

Post by marginalia_nu »

sethammons wrote: post #25
marginalia_nu wrote: post #19

Earlier quoted context omitted.

Threads isn't the only bottleneck in crawling. Assuming you're crawling at a civilized rate of about 1 request/second, you're only so many network hiccups away from consuming the entire ephemeral port range with connections in TIME_WAIT or CLOSE_WAIT.

Crank up the ulimit. And what is this 1 req/second nonsense? 1 req/sec/domain maybe. I have to agree, my first thought is "why this is not a single node in Go?"

Oh yeah, I mean 1 req/sec per domain of course.

It's very easy to end up with tens of thousands of not-quite-closed connections while crawling, even with SO_LINGER(0), proper closure, tweaking the TCP settings, and doing all things by the book.

It's a different situation from e.g. having a bunch of incoming connections to a node, the traffic patterns are not at all similar.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#27

Post by bluedino »

samwillis wrote: post #8

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…

> Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources.

Instead of setting up some kind of partnership with our vendors, where they just send us information or provide an API, we scrape their websites.

The old version ran in a hour, using one thread on one machine. Downloaded PDF's and extracted the values.

The new version is Selenium based, uses 20 cores, 300GB of memory, and takes all night to run. It does the same thing, but from inside of a browser.

As a bonus, the 'web scrapers' were blamed for every performance issue we had for a long time.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#29

Post by tempest_ »

samwillis wrote: post #8

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…

Site owners adding browser fingerprinting explicitly is vanishingly rare however site owners who sit behind cloud flare who very likely fingerprint browsers is very common.
Post reply on HN