What AI means for content creators: how to use it and how to fight it

lucas_crocker wrote: post #3

Also, does anyone have anything I missed? I know some people were suggesting not letting google index everything by updating your robot.txt file.

robots.txt isn't an ideal way of preventing pages being indexed.

X-robots http headers are more reliable: https://developers.google.com/search/docs/crawling-indexing/...

Regarding AI, it's a bit more tricky since it isn't going to abide by your rules. Cloudflare have tools: https://blog.cloudflare.com/ai-bots/

How effective these are though, IDK?

Re: What AI means for content creators: how to use it and how to fight it

#5

Post by lucas_crocker » Sat, May 18, 2024, 3:38 PM UTC

dazc wrote: post #4

lucas_crocker wrote: post #3

Also, does anyone have anything I missed? I know some people were suggesting not letting google index everything by updating your robot.txt file.

robots.txt isn't an ideal way of preventing pages being indexed. X-robots http headers are more reliable: https://developers.google.com/search/docs/crawling-indexing/... Regarding AI, it's a bit more tricky since it isn't going to abide by your rules. Cloudflare have tools: https://blog.cloudflare.com/ai-bots/ How effective these are though, IDK?

Oh that's interesting. Yea, paywalls might end being the most effective way of preventing AI bots, but even those could be circumvented. It's a tricky problem for sure.