Why not use robots.txt instead of littering your html with googlebot instructions?
Or even better, iptables rules :P
Doesn’t that mean you have to know every IP Address used by Google bot now and in the future?
Removing my site from Google search
11–20 of 119 posts
Re: Removing my site from Google search
#12seems like it would hurt your traffic
Re: Removing my site from Google search
#13Why not use robots.txt instead of littering your html with googlebot instructions?
Hi, author here. Google stopped supporting robots.txt [edit: as a way to fully remove your site] a few years ago, so these meta tags are now the recommended way of keeping their crawler at bay: https://developers.google.com/search/blog/2019/07/a-note-on-...
Did you think that mighty Google would pay attention to your puny "noindex" tag? Ha!
Re: Removing my site from Google search
#14seems like it would hurt your traffic
Just join a web ring like in the old days.
Re: Removing my site from Google search
#15seems like it would hurt your traffic
[deleted]
Re: Removing my site from Google search
#16seems like it would hurt your traffic
That depends on where your traffic originates from. Back when I tracked people on my site, I found I got very little from search results. Most of it (> 95%) came from links from social media and Github. On a blog that's heavily about privacy I wouldn't expect much to come from Google.
Also, so what if the numbers go down? If your reason from writing a blog is to see a number on a screen then what does that actually give you?
Re: Removing my site from Google search
#17Oh hey I thought I was the only one. lucb1e.com and another site are also not indexed, though I blocked it based on the user agent string. That way it doesn't get page data or non-HTML files from my server. I introduced this when they were pulling this AMP thing: https://lucb1e.com/?p=post&id=130 It personally doesn't impact me, but it impacts other people on the internet and I figured it was the only thing I can do to try and diversify this market (since I myself already switched to another search engine).
There are zero other restrictions on my site. Use any search engine other than google. Or don't, up to you.
Re: Removing my site from Google search
#18Earlier quoted context omitted.
Or even better, iptables rules :P
Doesn’t that mean you have to know every IP Address used by Google bot now and in the future?
Not a very hard problem; after all, many websites allow full access to Googlebot IP ranges yet show a paywall to everyone else (including competing search engines).
I also happen to ban Google ranges on multiple less-public sites specially since they completely ignore robots.txt and crawl-delay.
Re: Removing my site from Google search
#19Earlier quoted context omitted.
Hi, author here. Google stopped supporting robots.txt [edit: as a way to fully remove your site] a few years ago, so these meta tags are now the recommended way of keeping their crawler at bay: https://developers.google.com/search/blog/2019/07/a-note-on-...
Did you think that mighty Google would pay attention to your puny "noindex" tag? Ha!
[deleted]
Re: Removing my site from Google search
#20Earlier quoted context omitted.
Or even better, iptables rules :P
Doesn’t that mean you have to know every IP Address used by Google bot now and in the future?
The way to check googlebot (in a way that will be resistant to expansion of Googlebot's IP ranges in the future) is to perform hostname lookup, with dns lookup as well to verify that the rDNS isn't a lie: https://developers.google.com/search/docs/advanced/crawling/...