101–110 of 342 posts
It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defending against AI parasitism.
this just pushes AI agents "underground" to adopt the behavior of a full blown stealth focused scraper which makes it harder to detect.
At Coinbase, we've been building tools to make the blockchain the ideal payment rails for use cases like this with our x402 protocol:
Ping if you're interested in joining our open source community.
I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference.…
Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…
I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # wr…
"User-agent: CCBot disallow: /" Is Common Crawl exclusively for "AI" CCBot was already in so many robots.txt prior to this How is CC supposed to know or control how people use the archive contents What if CC is relying on fair use # To request permission to license our intellectual # property andd/or other materials, please contact this # site's operator directly If the operator has no intellectual property rights in…
# To request permission to license our intellectual
# property andd/or other materials, please contact this
# site's operator directly
Ironically, I've only ever scraped sites that block CCBot, otherwise I'd rather go to Common Crawl for the data.
Earlier quoted context omitted.
It comes down to: 1. Use the Cache-Control header to express how to cache your site correctly ( https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cac... ) 2. Use a CDN service, or at least a caching reverse proxy, to serve most of the cacheable requests to reduce load on the (typically much more expensive) origin servers
Just note that many AI scrapers will go to great length to do cache busting. For some reason many of them feel like they need to get the absolute latest version and don't trust your cache.
We are in the Napster phase of Web content stealing.
The list of bots is pretty short right now: https://developers.cloudflare.com/bots/concepts/bot/#ai-bots
> AI bots > You can opt into a managed rule that will block bots that we categorize as artificial intelligence (AI) crawlers (“AI Bots”) from visiting your website. Customers may choose to do this to prevent AI-related usage of their content, such as training large language models (LLM). > CCBot (Common Crawl) Common Crawl is not an AI bot: https://commoncrawl.org
Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…