Crawling a quarter billion webpages in 40 hours (2012)

[flagged]

don't want to release the source code because of moral reservations about how people might use it. Ok upstanding citizen.

Nothing to stop you writing a similar project and releasing it if you feel so strongly that it should be out there, instead of being incensed by not being give then sweat off someone else's brow.

This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping.

You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this:

- SPAs with front end only rendering are hard to scrape with a traditional http library.

- Anti bot/scraping technology fingerprints browsers, looks at request patterns, and browser behaviour to try and detect and block bots.

Using a "real browser" is often advised as a way around these issues.

However from my experience you should avoid headless browser crawling until it is absolutely necessary, I have found:

- Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources.

- Most SPAs now have some level of server side rendering, and often that includes having a handy JSON in the returned document that contains the data you actually want.

- Advanced browser fingerprinting is vanishingly rare. At most I have seen detection of user agent strings and comparing them to http headers and the order of them. If you make your http lib look like a current browser you are %99.9 of the way there.

It also depends on what the goal is. If you need to extract some data from a specific site (as opposed to obtaining faithful representations of pages from arbitrary sites like a search engine might need to), then SPAs might be the easiest targets of all. Just do some inspection beforehand to find out where the SPA is loading the data from, very often the easiest thing is to query the API directly.

Sometimes the API will need some form of session tokens found in HTTP meta tags, form inputs, or cookies. Sometimes they have some CORS checks in place that are very easy to bypass from a server-side environment by spoofing the Origin header.