There's an easier way - don't implement AMP.
Dear Google, I'm Blocking You from My Website
21–30 of 215 posts
Re: Dear Google, I'm Blocking You from My Website
#22> As fewer results of such not-for-profit websites show up in Google Search, competition will be reintroduced to the search market. I don't think it will. While I personally hate AMP, and disagree with many of Google's practices, this is not the solution. Google is everywhere . It is the default search engine on almost every browser, as well as on most devices. How often do you hear someone say "look it up" as oppose…
I'm not saying it proves anything, but Kleenex is synonymous with facial tissue and Velcro is synonymous with hook and loop fasteners. I "google" things on DuckDuckGo. :)
Re: Dear Google, I'm Blocking You from My Website
#23> Google needs to stick to being a solid search engine instead of trying to be gate keepers While I agree with the sentiment, I feel like this ship sailed years ago - it's like saying "Amazon should stick to being an ecommerce platform". What's more, Google was never in the business of "being a solid search engine". Google is and always has been in the business of selling access to users' eyeballs. AMP - as much as I…
> What's more, Google was never in the business of "being a solid search engine". Google is and always has been in the business of selling access to users' eyeballs.
That's not how they became the darling of the tech scene though.
http://infolab.stanford.edu/~backrub/google.html
> Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users [..] we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers. Since it is very difficult even for experts to evaluate search engines, search engine bias is particularly insidious. [..] we believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm.
Re: Dear Google, I'm Blocking You from My Website
#24Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive. Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are remov…
How do you block their IP? Google has hundreds of numbers and AFAIK they don’t even publish all of them.
Verifying Googlebot: https://support.google.com/webmasters/answer/80553?hl=en
Re: Dear Google, I'm Blocking You from My Website
#25yeah but then how do you get organic traffic?
On a related note, what does organic search traffic look like these days? I was once personally invested in running several small content sites and at the time organic traffic was pretty much entirely from Google. There were other inbound sources - Bing, Yahoo!, etc. - but they were so negligible that it would have been reasonable to ignore them completely for the purposes of optimization. I've been using DDG as my p…
I agree, I use Startpage.com
That said, looks like most of the US does not feel the same way about Google. I still see 90-99% of organic traffic coming from Google for my client sites.
Re: Dear Google, I'm Blocking You from My Website
#26This site loads amazingly fast! Proves the point that you really don't need AMP.
You do if you want Google to send any traffic to it. Your only option is to bend the knee unless you're site is nothing more than a blog which you do for fun.
Re: Dear Google, I'm Blocking You from My Website
#27Earlier quoted context omitted.
How do you block their IP? Google has hundreds of numbers and AFAIK they don’t even publish all of them.
There are services that track google IPs, those doing black hat SEO and cloaking rely on them. Google is going to crawl, you just can’t stop them from crawling even if you tell them not to in robots.txt. Best option at this point is to add a noindex tag on pages and let them crawl to see it.
Wouldn't that be considered illegal due to the same reasons as scraping?
Re: Dear Google, I'm Blocking You from My Website
#28Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive. Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are remov…
Should read: 'Dear Google, I'm giving you full access to my website but I'm asking you not to index it.'
Re: Dear Google, I'm Blocking You from My Website
#29Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive. Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are remov…
How do you block their IP? Google has hundreds of numbers and AFAIK they don’t even publish all of them.
As others mentioned, you can block their user-agent. That will stop most crawling. They have some bots that use browser agents to see if you serve different content to non-google bots.
To block those, you can feed their IP's to haproxy or iptables ipset.
fetch_google()
{
for line in $(dig +short txt _cloud-netblocks.googleusercontent.com | tr " " "\n" | grep include | cut -f 2 -d :)
do
dig +short txt "${line}"
done | tr " " "\n" | grep ip4 | cut -f 2 -d : | sort -n | uniq | xz -9ecv > _GOOGLE.netset.xz
};
acl BOTS src -f bot.txt
Re: Dear Google, I'm Blocking You from My Website
#30Is it possible to make an AMP page that just redirects to a non-AMP one?