I built a flash crawler to index all Flash while at Adobe. It started with Alexa top 1M I think then crawled. This was 2008-2010 I think so we had to do a lot of custom stuff, but we basically crawled then ran a headless Firefox with a custom headless Flash player that dumped a ton of data so also analyzed every flash at runtime and indexed all of that. We built a dedicated cluster in a colocation center in Bucharest…
But everyone's in an AWS world right now.