The Evolution of Marginalia's Crawling

I'd be curious if the new architecture can be adapted to use different content types, other than HTML. I think the Fediverse (even if it can be crawled as the normal web pages) it would benefit for a custom crawler that can jump from inbox to inbox and understands (at least some) ActivityPub jsonld.

Yeah, that should be fairly doable.

The beauty of the design, since it's mediated by a portable language of JSON-objects, is that you can in principle replace or extend any of the steps, including support for other protocols (like gemini:// ?) or content-types.

It's very simple to write a basic ActivityPub crawler (I have done this), but you'd go from outbox to outbox, not inbox to inbox. Regardless, it's all just JSON, so it is more straightforward to crawl than the web.

I tried writing a crawler to map the Fediverse (to discover homeservers) but I discovered quickly that most of the content in the Fediverse is extremist content, both left-wing and right-wing -- it's basically all tankies and Nazis, with a few techies in a bubble using mastodon.social, who think Fediblock is a solution to this problem.

Oh right, and there are a ton of pedo instances too, mostly based in Japan where "loli" is legal. Due to the way ActivityPub publishes federated content, I didn't feel comfortable running a homeserver, because a pedo instance could federate illegal content to my homeserver and get me arrested and charged for possession of content I did not request and do not want to store on my computers, and if I'm not monitoring what content is getting federated to my timeline, I might not even know it's there. Too risky.

Demoralized, I abandoned the project, and I don't think much about ActivityPub anymore.

I've started to build a search engine as a hobby side project (I'm three weekends in. It's very much inspired by marginalia, but with different content, curation, and UX goals. I have beefy hardware at home that can see it through.) My early crawl architecture plans actually mirror this new scheme, so it's delightful to have a bit of confirmation I'm on a mildly correct track.

One of my long-running side projects has been to do a lot of web-scraping on a website with continually evolving content, that's related to the economics of a side business, and yeah—we hit a lot of those growing pains of a monolithic scraper (poor debugging, unexplained re-crawling, waiting too long for a crawl plan, rare race conditions w/ multiple scrapers), averted by this model.

I kinda want to know how the big G does their crawling—I watch all the different googlebots hit my many websites, but I'm curious to see their architecture of prioritization and analysis—but a lot of the analysis is definitely the resulting sauce that should be kept somewhat proprietary and secret.

I've noted my search engine index starts to get noticeably stale after about two months, and right now it takes 2 or so weeks to crawl. Extrapolating, that might mean that it's sustainable up to about 4x the size of my current index.

Right now a lot of this work is manual. Kinda not long-term viable, but for now it's not too bad. Kick off a script every now and again and come back a week later and see how it went.

After that, there needs to be some sort of prioritization. Maybe a job that randomly probes URLs and flags dead/changed links for priority re-crawling. I think using RSS feeds to detect new links is also a tool that might be useful.