I came up with the idea and helped build Grub, the distributed crawler. Looksmart bought it, ran it for a time, then sold it to Wikimedia. I reclaimed the name recently (abandoned mark) and have a new crawler now that is agentic. I use it for my own research runs, and it's not my main focus at this point, nor am I trying to get it attention. A lot of LLMs and coding agents can easily fetch content if it is needed and we're blind to how they do it. See the Claude plugin for Chrome as an example of using it in a user-in-the-loop solution. That said, I've been spending a little time thinking about how to bake the contract into the crawler, as opposed to expecting someone else to act ethically using it.
Grub was, in a very real way, a botnet. And, we harmed site owners when we were operating at full capacity. There were a few bugs in the early days where we would reschedule a site because the ingestion in the server broke, which then caused the page to be rescheduled. Stupid error, and we fixed it, but it's illustrative of the fact even good intentions isn't enough here.
What I've come up with over the years is similar to the idea Cloudflare is implementing with payments to site owners by charging the crawlers. My objection to Cloudflare's implementation is based on a personal opinion about Cloudflare being a single point of failure and also a decrypted choke point. Their ideas about how to handle crawlers, and pay for the load on the sites is solid. It presumes to use the 402 response to demand payment. I'm clearly biased about Cloudflare, but that's my prerogative here.
It may be possible to solve this with cryptocurrency, in a distributed way, and I've prototyped a system that uses the Lightning Network to handle the payments from a 402 response. Lightning Labs also worked on a project called Apeture for a time that did something similar.
HN's site knows every item ID, and it knows fresh IDs get read in a predictable distribution while old ones mostly sleep. Sustained access outside that is itself the scraper signal. No IP reputation needed, which matters now that residential proxies burn an address after a handful of requests.
Karma gives you a clean way to let humans through. Issue logged-in accounts with decent karma a token whose cold-content budget scales with it (the karma), so an account with history scrolling back through a 2014 thread just reads it. Karma should gate the tier, not be spent as currency, or upvote rings become a crawling business.
Anonymous readers who deep link into one old thread from a search engine get the first fetch or two free (and you watch the article IDs, not the IPs). What remains after those carve-outs is bulk traversal of cold IDs with no identity attached, and that traffic gets rate limited and answered with a 402: pay per page over Lightning, priced at a healthy multiple of what residential proxy bandwidth already costs, or come back slowly for free.
There are probably holes in these thoughts. It's one of the harder problems to solve, for sure.