Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to stop Google from AI-summarising your website

teruza.com

Post reply on HN
Print view View original

11–20 of 78 posts

Re: How to stop Google from AI-summarising your website

#11

Post by muppetman »

I have this in my Apache conf for a site I don't want indexed/archived etc.

Header set X-Robots-Tag "noindex, nofollow, noarchive, nositelinkssearchbox, nosnippet, notranslate, noimageindex"

Of course, only the beeping Internet Archive totally ignored it and scraped my site. And now, despite me trying many times, they won't remove it.

It seems to mostly work, I also have Anubis in front of it now to keep the scrapers at bay.

(It's a personal diary website, started in 2000 before the term "blog" existed [EDIT: Not true - see below comment]. I know it's public content, I just don't want it searchable public)

Re: How to stop Google from AI-summarising your website

#12

Post by victorbjorklund »

pupppet wrote: post #7

I don't understand how these AI summaries don't cannibalize Google's future profits. Google lives off ads that direct users to websites, websites they are doing their damnedest to make unnecessary. Who will be building future websites that nobody visits.

They make 99% of their profits on high-intent searches like "buy macbook" or "book trip to dc". They make much less on informational searches like "how to fix cors error on javascript" (most likely they make zero on it)

Re: How to stop Google from AI-summarising your website

#13

Post by bayindirh »

muppetman wrote: post #11

I have this in my Apache conf for a site I don't want indexed/archived etc. Header set X-Robots-Tag "noindex, nofollow, noarchive, nositelinkssearchbox, nosnippet, notranslate, noimageindex" Of course, only the beeping Internet Archive totally ignored it and scraped my site. And now, despite me trying many times, they won't remove it. It seems to mostly work, I also have Anubis in front of it now to keep the scrapers…

I have recently found out that the snapshots have a "why?" field. The archivers might not be internet archive themselves, but commoncrawl, archive team, etc. pushing your site to Internet Archive.

Look at the reason, and get mad to the correct people.

It might be the archive themselves, but just be sure.

Re: How to stop Google from AI-summarising your website

#14

Post by IcyWindows »

So only the rich can hire humans to speed up searching by viewing each page and summarizing the content for their employer?

This feels like the wrong solution for wanting to be compensated for information.

I don't how what the solution is because one often doesn't know if the information is worth paying for until after viewing it.

Re: How to stop Google from AI-summarising your website

#15

Post by bayindirh »

pupppet wrote: post #7

I don't understand how these AI summaries don't cannibalize Google's future profits. Google lives off ads that direct users to websites, websites they are doing their damnedest to make unnecessary. Who will be building future websites that nobody visits.

Because they also have a tech where AI-Agents can add product and service advertisements into these summaries [0].

They won an award for the paper, and the example they given was a "holiday" search, where a hotel inserted their name, and an airline company wedged themselves as the best way to go there.

If I can find it again, I'll print and stick its link all over walls to make sure everybody knows what Google is up to.

Edit: Found it!

[0]: https://research.google/blog/mechanism-design-for-large-lang...

Re: How to stop Google from AI-summarising your website

#17

Post by blueg3 »

muppetman wrote: post #11

I have this in my Apache conf for a site I don't want indexed/archived etc. Header set X-Robots-Tag "noindex, nofollow, noarchive, nositelinkssearchbox, nosnippet, notranslate, noimageindex" Of course, only the beeping Internet Archive totally ignored it and scraped my site. And now, despite me trying many times, they won't remove it. It seems to mostly work, I also have Anubis in front of it now to keep the scrapers…

The term blog existed in 1999, and "weblog" in 97.

Re: How to stop Google from AI-summarising your website

#18

Post by hombre_fatal »

pupppet wrote: post #7

I don't understand how these AI summaries don't cannibalize Google's future profits. Google lives off ads that direct users to websites, websites they are doing their damnedest to make unnecessary. Who will be building future websites that nobody visits.

I'm sure they added it with reluctance, and they had to do it because LLM services are eating Google Search's lunch.

Google even put the AI snippet above their ads, so you know how bad it stings.

Re: How to stop Google from AI-summarising your website

#19

Post by ozaark »

raincole wrote: post #8

Title: > and Reclaim Your Organic Traffic Content: > 1. Set Snippet Length to Zero with max-snippet:0 Sure, buddy, sure. Users are notorious for clicking a link in search result without description, right.

I believe max-snippet removes suggested text from the SERPs but would still display the page meta description as per usual.

Re: How to stop Google from AI-summarising your website

#20

Post by asdefghyk »

muppetman wrote: post #11

I have this in my Apache conf for a site I don't want indexed/archived etc. Header set X-Robots-Tag "noindex, nofollow, noarchive, nositelinkssearchbox, nosnippet, notranslate, noimageindex" Of course, only the beeping Internet Archive totally ignored it and scraped my site. And now, despite me trying many times, they won't remove it. It seems to mostly work, I also have Anubis in front of it now to keep the scrapers…

RE "...Of course, only the beeping Internet Archive totally ignored it and scraped my site. And now, despite me trying many times, they won't remove it...."

Why would you NOT want internet archive to scrape your website? (Im Clueless - thank you)

Post reply on HN