Pulling my site from Google over AI training
tracydurnell.com
Pulling my site from Google over AI training
1–10 of 102 posts
Re: Pulling my site from Google over AI training
#2Helpful links. I will be doing the same this evening
Re: Pulling my site from Google over AI training
#3> I’m going to start by pulling my websites out of Google search, then work on adding my sites to directories. Maybe I’ll even join a webring
I'm curious, this is the first time I've heard of a webring, I'd like to learn more about these alternate discovery routes. Anyone have any concrete experience or recommendations to share?
Re: Pulling my site from Google over AI training
#4> Blocking bots that collect training data for AIs (and more)
> In addition, I created a robots.txt file to tell “law abiding” bots what they’re not allowed to look at. I ought to have done this before but kind of assumed it came with my WordPress install (Nope.)
> I specifically want to deter my website being used for training LLMs, so I blocked Common Crawl.
Instead of blocking, it would be neater to present and alternative version to the crawlers (like many paywalled sites already to for SEO) that's full of dynamically generated LLM-generated garbage. That'll help the LLMs poison themselves.
Re: Pulling my site from Google over AI training
#5> I’m going to start by pulling my websites out of Google search, then work on adding my sites to directories. Maybe I’ll even join a webring I'm curious, this is the first time I've heard of a webring, I'd like to learn more about these alternate discovery routes. Anyone have any concrete experience or recommendations to share?
Directory sites and webrings were artifacts of the early web, when search engines were pretty bad. They disappeared once search got better.
Re: Pulling my site from Google over AI training
#6An interesting thought are government or foreign actors training a generative Ai. They won't abide by any no-index tax and will scrape any and everything.
Re: Pulling my site from Google over AI training
#7Author should update their robots.txt for Googlebot as well. It is not clear if noindex means "notrain" too. The entire webpage has to be read in and parsed for Google to extract that meta tag. However, robots.txt should stop the crawler before it proceeds to the rest of your site.
Re: Pulling my site from Google over AI training
#8> I’m going to start by pulling my websites out of Google search, then work on adding my sites to directories. Maybe I’ll even join a webring I'm curious, this is the first time I've heard of a webring, I'd like to learn more about these alternate discovery routes. Anyone have any concrete experience or recommendations to share?
Directory sites and webrings were artifacts of the early web, when search engines were pretty bad. They disappeared once search got better. https://en.wikipedia.org/wiki/Webring
I miss webrings. I don't think that search getting better killed them (they are useful for reasons unrelated to the state of search), but when personal and hobbyist websites started vanishing, webrings went with them.
Re: Pulling my site from Google over AI training
#9> Blocking bots that collect training data for AIs (and more) > In addition, I created a robots.txt file to tell “law abiding” bots what they’re not allowed to look at. I ought to have done this before but kind of assumed it came with my WordPress install (Nope.) > I specifically want to deter my website being used for training LLMs, so I blocked Common Crawl. Instead of blocking, it would be neater to present and al…
I was thinking about jamming along these lines, but the problem is that it's a game of whack-a-mole -- you have to keep up on what bots are active (robots.txt doesn't really help here, and focusing on Common Crawl is insufficient).
My websites have been closed to the public since shortly after the release of ChatGPT, but I've been considering opening them up again, sort of. The not-logged-in experience being full of dynamically generated LLM poison as you suggest -- for everybody rather than trying to single out crawlers -- and you have to log in to get to the real contents of the site.
Re: Pulling my site from Google over AI training
#10Can you pollute their data with hidden elements, or do they only scrape visible stuff?