Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Crawling a quarter billion webpages in 40 hours (2012)

michaelnielsen.org

Post reply on HN
Print view View original

11–20 of 68 posts

Re: Crawling a quarter billion webpages in 40 hours (2012)

#11

Post by Nakroma »

samwillis wrote: post #8

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…

I do a lot of scraping at my day job, and I agree the times we need to use a headless browser are very rare. Vast majority of the time you can either find an API or JSON endpoint, or scrape the JS contained in the returned SSR document to get what you want.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#12

Post by londons_explore »

samwillis wrote: post #8

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…

Headless browsers are expensive, but if you needed them, I suspect there are easy wins to be had on the performance front.

For example, you can probably skip all layout/rendering work and simply return faked values whenever javascript tries to read back a canvas it just drew into, or tries to read the computed width of some element. The vast majority of those things won't prevent the page loading.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#13

Post by marginalia_nu »

samwillis wrote: post #8

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…

Depends on what you want with the data I guess. From a search engine point of view, the SPA case isn't that relevant, since SPAs can't reliably be linked to and in general tend to not be very stable and it's overall difficult to figure out how to enumerate and traverse their views.

I think a good middle ground might be to do a first pass with a "stupid" crawler, and then re-visit the sites where you were blocked or that just contained a bunch of javascript with a headless browser.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#17

Post by ricardo81 »

spacebacon wrote: post #9

How to spend $580 in 40 hours. More can be done for much less in 2012.

Yes. I used lib curl's multi interface on one $40/m server around that time. Indeed at any scale the rate limiting becomes the main bottleneck, mainly because a lot of sites are concentrated on certain hosts. Speed isn't the problem and multiple servers aren't really needed.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#18

Post by abhibeckert »

marginalia_nu wrote: post #13
samwillis wrote: post #8

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…

Depends on what you want with the data I guess. From a search engine point of view, the SPA case isn't that relevant, since SPAs can't reliably be linked to and in general tend to not be very stable and it's overall difficult to figure out how to enumerate and traverse their views. I think a good middle ground might be to do a first pass with a "stupid" crawler, and then re-visit the sites where you were blocked or t…

The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content.

If Google can crawl it, then you can too. And while Google doesn't use a headless browser (or at least I assume they don't) they absolutely do execute javascript before loading the content of the page. And they execute the click event handlers on every link/button and when we use "history.pushState()" to change the URL Google considers that a new page.

You're just going to get a loading spinner with no content if you do a dumb crawl (I disagree with that and think we should be running a headless browser server side to execute javascript and generate the initial page content for all our pages... but so far management hasn't prioritised that change... instead they just keep telling us to make our client side javascript run faster... imagine if there was no javascript to execute at all? At least none before first contentfull paint)

Re: Crawling a quarter billion webpages in 40 hours (2012)

#19

Post by marginalia_nu »

ddorian43 wrote: post #16

With async in any language of choice this should need just 1 server. 250M/40hours=1.7K requests/second. You can probably do it on a single core in a low level language like Rust/Java.

Threads isn't the only bottleneck in crawling.

Assuming you're crawling at a civilized rate of about 1 request/second, you're only so many network hiccups away from consuming the entire ephemeral port range with connections in TIME_WAIT or CLOSE_WAIT.

Re: Crawling a quarter billion webpages in 40 hours (2012)

#20

Post by ricardo81 »

samwillis wrote: post #8

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…

The issues you list can probably be split into 2 main issues

- The host has a tendency to block unknown user agents, or user agents that claim to be a browser but are not

- Anything that requires client side rendering

I'd suppose both problems are more pertinent in 2023 than they were in 2012.

At web scale, the issue appears on at what point would you be required to use a headless browser when not using one in the first place e.g. if React is included/referenced. Perhaps some simple fingerprinting of JS files would do, IMO in reality the line is very blurry, so you either do or don't.

Post reply on HN