11–20 of 215 posts
I’ve made the argument before that Google is the new AOL homepage. It’s the gateway to the web for a huge majority as AOL was at one time. AMP is concentrating Google’s web just as AOL once did for first timers.
Google has much more utility but it’s still very similar. AOL had email, web, chat, news, shopping? (maybe late days of AOL), and let’s not forget search.
Are they actually blocking Google somehow? No robots.txt, no 403 when using a Googlebot user agent to fetch the site page...
yeah but then how do you get organic traffic?
I've been using DDG as my primary search engine now for several years, and using Google no longer "feels like the Web" for me. I imagine I have one hell of a case of observation bias in that, but it does seem like alternatives have been gaining some traction in recent years.
Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive. Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are remov…
Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive. Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are remov…
How do you block their IP? Google has hundreds of numbers and AFAIK they don’t even publish all of them.
Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive. Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are remov…
I deeply hate AMP. It's a monopolistic move to bully companies into handing over Google their content.
Are they actually blocking Google somehow? No robots.txt, no 403 when using a Googlebot user agent to fetch the site page...
Googlebot looks for this meta tag.
Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive. Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are remov…
How do you block their IP? Google has hundreds of numbers and AFAIK they don’t even publish all of them.
Google is going to crawl, you just can’t stop them from crawling even if you tell them not to in robots.txt. Best option at this point is to add a noindex tag on pages and let them crawl to see it.