Just use 500 android phones running Tmux they said. It'll be easy they said.
Imagine a Beowulf cluster of android phones running tmux!
Crawling a quarter billion webpages in 40 hours (2012)
31–40 of 68 posts
Re: Crawling a quarter billion webpages in 40 hours (2012)
#32Earlier quoted context omitted.
The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content. If Google can crawl it, then you can too. And while Google doesn't use a headless browser (or at least I assume they don't) they absolutely do execute javascript before loading the conte…
> The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content. This is true for some SPAs, but not all SPAs, and there's not really any way of telling which is which. I don't personally attempt to crawl SPAs because it's not the sort of content I…
I have a pet theory that there are two forms of the web: the document web and the application web. SPAs have some very attractive properties for the application web but complicate/break the document web.
That being said, with sites like HN, Reddit, LinkedIn, Twitter, news outlets, etc. the lines between “document” and “application” get blurred. In some ways they’ve built a micro-application that hosts documents. Content can be user submitted in-browser. Content can be “engaged with” in browser. Some handle this blurring better than others. HN is an example IMO of getting it right where nearly everything that should be addressable (like comments) can be linked to. Others not so much.
(As an aside, I love marginalia!)
Re: Crawling a quarter billion webpages in 40 hours (2012)
#33This is a good overview, but being from 2012 it's missing comment on one now common area, using a real browser for crawling/scraping. You will often now hear recommendations to use a real browser, in headless mode, for crawling. There are two reasons for this: - SPAs with front end only rendering are hard to scrape with a traditional http library. - Anti bot/scraping technology fingerprints browsers, looks at request…
Is there like a custom performance based headless browser
How are they scaled?
Re: Crawling a quarter billion webpages in 40 hours (2012)
#34Re: Crawling a quarter billion webpages in 40 hours (2012)
#35Re: Crawling a quarter billion webpages in 40 hours (2012)
#36Earlier quoted context omitted.
> The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content. This is true for some SPAs, but not all SPAs, and there's not really any way of telling which is which. I don't personally attempt to crawl SPAs because it's not the sort of content I…
I have a pet theory that there are two forms of the web: the document web and the application web. SPAs have some very attractive properties for the application web but complicate/break the document web. That being said, with sites like HN, Reddit, LinkedIn, Twitter, news outlets, etc. the lines between “document” and “application” get blurred. In some ways they’ve built a micro-application that hosts documents. Cont…
For application websites like the ones you listed, you'd typically end up building a special integration for crawling against their API or data dumps. This is also true for github, stackoverflow, and even document:y websites like wikipedia.
It's simply not feasible to treat them as any other website if you wanna index their data.
Re: Crawling a quarter billion webpages in 40 hours (2012)
#37Archive link: https://archive.is/yUWjh Also kinda wish the author paid any sort of attention to the fact that doing this incorrectly may create a flood of DNS queries. At least have the decency to set up a bind cache or something.
what is a bind cache? always assume most of us have terrible knowledge of networking
Re: Crawling a quarter billion webpages in 40 hours (2012)
#38a bit of context - posted this because i found it via https://twitter.com/willdepue/status/1669074758208749572
if they replicate it in 2023 it would be pretty interesting to me. i can think of a few times a year i need a good scraper.
but also thought it a good look into 2012 Michael Nielsen, and into thinking about performance.
Re: Crawling a quarter billion webpages in 40 hours (2012)
#39How to get the top million sites list today? Alexa has shifted focus recently.
Check out Tranco [1], which uses Cisco Umbrella, Majestic and also now a list sourced from Farsight passive DNS [2]. They're "working on" adding Chrome UX and Cloudflare Radar.
There's also a list from Netcraft [3].
[2] https://www.domaintools.com/resources/blog/mirror-mirror-on-...
Re: Crawling a quarter billion webpages in 40 hours (2012)
#40A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results, only to find out later that a large CDN provider has identified me as a spammy client with their client reputation score and blocked my IP address on all of the websites that they serve.
Changing your IP address with AT&T is a pain (even though the claim is that your IP is dynamic, in practice it hardly ever changes) so I opted to contact the CDN vendor by filling out a form and luckily my ban was lifted in a day or two. Nevertheless, it was annoying that suddenly a quarter of the websites I normally visit were not accessible to me since the CDN covers a large swath of the Internet.