A year of fighting scrapers on my 1.5 million-page website

Is there some existing mechanism already that counts how often an ip only scrapes the page and not the css and then block those origin IPs if it occurs “too often”?

Unfortunately, the best practice is to make css cacheable so you need to keep long histories.

Previously, I had done a fair amount of research into how Google's monopoly on web crawling further entrenches their monopoly in the search engine market. You can read more about this here, https://knuckleheads.club, there is a long report from ~2020 or so that explains how it worked at the time. The club is mothballed, I am doing other things with my life, and I'm happy to say that we played a very small role in the DOJ ordering Google to share their crawl data with qualified competitors (a work in progress, but it's progressing).

Chatbots have super charged this dynamic though, to the point that it is showing up in the robots.txt data. The last few weeks I've been having Claude rerun some old analysis of Common Crawl from back then, when I have spare usage and time. What I've found is that you can see pretty clearly the rise in people outright blocking AI chatbot related crawlers likely because of how aggressive they have become.

  Quarter  Crawl    GPT     Claude  CC      G-Ext   Byte    Bing    Google
  2023 Q1  2023-06  0.00%   0.00%   0.16%   0.00%   0.06%   0.47%   0.39%
  2023 Q2  2023-14  0.00%   0.00%   0.18%   0.00%   0.06%   0.45%   0.38%
  2023 Q3  none       —       —       —       —       —       —       —
  2023 Q4  2023-40  2.21%   0.00%   2.12%   0.04%   0.11%   0.39%   0.27%
  2024 Q1  2024-10  0.53%   0.05%   0.31%   0.09%   0.18%   0.34%   0.31%
  2024 Q2  2024-18  0.55%   0.09%   0.32%   0.11%   0.24%   0.32%   0.31%
  2024 Q3  2024-30  0.68%   0.22%   0.36%   0.20%   0.38%   0.24%   0.33%
  2024 Q4  2024-42  1.10%   0.50%   0.44%   0.32%   0.50%   0.25%   0.40%
  2025 Q1  2025-05  1.14%   0.66%   0.54%   0.42%   0.66%   0.25%   0.44%
  2025 Q2  2025-18  1.37%   0.93%   0.63%   0.70%   0.92%   0.29%   0.19%
  2025 Q3  2025-30  1.42%   1.07%   0.74%   0.62%   1.01%   0.31%   0.27%
  2025 Q4  2025-43  1.92%   1.51%   1.23%   1.15%   1.52%   0.27%   0.19%
  2026 Q1  2026-04  2.13%   1.76%   1.68%   1.58%   1.77%   0.22%   0.15%
  2026 Q2  2026-17  2.80%   2.38%   2.26%   2.13%   2.50%   0.22%   0.14%
  2026 Q3  2026-30  3.45%   3.01%   2.89%   2.71%   3.16%   0.21%   0.14%

GPTBot is OpenAI, ClaudeBot is Anthropic, CCBot is Common Crawl, Google-Ext is a way for website owners to indicate they don't want their content to be used for AI, Bytespider is Bytedance, Bing and Google are the last two. Take these numbers with a truck of salt, haven't had time to verify them.

It's very clear that website owners do not like getting their content scraped and are indicating to GPTBot et al. that they are not welcome. It's a shame that CCBot is caught in the cross fire, but that's life. Bing and Google are doing just fine though, almost like having significant power in the search engine market gives you an advantage in other markets too. Who knew!

It doesn't detect "real browsers". It simply adds some friction and is niche enough that AI scrapers have not bothered to bypass it yet.

I thought about that but to your point CSS is cachable. In fact I made mine immutable. No I just visually spot patterns and use that to study other facets of the agent, other client headers or lack thereof, supported protocol, accepted encoding and so on.

Bandwidth is still not free, if you cloud host.

I had a client ask me to look into why their bill spiked 5x (admittedly not by much - from a few dollars to tens). They run a static site, and it turned out someone had inadvertently replaced an image with a massive uncompressed photo. Every unique visitor was caching O(25MB) when they loaded the homepage. Also funny because now everyone has 100Mb+ internet, nobody noticed a few extra seconds on load.

There are lots of solutions to this if you actually need to serve a lot of bytes, but you can get stung even with very simple deployments. Not everyone wants to put Anubis or Cloudflare in front of their site.