71–80 of 119 posts
I use simple HTTP auth with an easy username and password on most my sites. It is rarely a problem for anyone I invite, except perhaps Instagram's browser, but no crawler traffic.
I'm sure someone at Google is pretty happy that they don't have to show this page in their search results. Nobody can accuse them of bias against anti-Google pages -- the site owner did it to themselves.
Seems like as perfect an example of "cutting off nose to spite face" as I can imagine. (ok, refusing the vax and dying of COVID to get back at the left might be a better example, but this one is close)
Earlier quoted context omitted.
Quote from the linked article: “ For those of you who relied on the noindex indexing directive in the robots.txt file, which controls crawling, there are a number of alternative options:” The first option is the meta tag. It does mention an alternative directive for robots.txt, however.
What about the blocking google bot by their IPs, also combined with user-agent wouldn't that stop the crawlers Google crawlers IPs https://www.lifewire.com/what-is-the-ip-address-of-google-81...
> If other pages point to your page with descriptive text, Google could still index the URL without visiting the page
Earlier quoted context omitted.
> How many visits does your blog have daily? ~200k sessions in the past year, so ~550/d. Breakdown: * ~30% search * ~30% no referer * ~25% HN * ~7% Twitter/FB/etc * ~8% other > what would it take you to remove your site from Google index? I don't see why I would want to exclude my site from any index? Being in indexes helps people find my writing, which I like!
> I don't see why I would want to exclude my site from any index? Being in indexes helps people find my writing, which I like! It's essentially a form of boycott. If one believes Google is a problematic entity (too many fingers in too many aspects of our lives), it's a way to sever connections with them at some personal cost. At least, if you care about search traffic - one might argue the assumption that Google-like…
There's already an interesting question about when boycotting is it worthwhile tactic, but in this situation, there's the additional complexity of there being two ends at which one could boycott a search engine:
* Producer: don't allow the search engine to include your stuff
* Consumer: use a different search engine
This is not the only place you find this dynamic. For example, if I thought Chrome was harmful, I could choose:
* Producer: make my site incompatible with Chrome and suggest people switch
* Consumer: use a different browser
Or, with email:
* Producer: don't email people with @gmail.com addresses
* Consumer: use a different email provider
Thinking through these cases and similar ones, if you think Gmail / Chrome / Search is harmful then the "consumer" side makes sense: the alternatives are nearly as good so you're not giving up much, and you're helping increase diversity. On the other hand, the "producer" side ones are much less attractive, because they're a much larger sacrifice and the benefit doesn't seem that big.
(Disclosure: I work at Google, but not on Search, Chrome, or Gmail)
Oh hey I thought I was the only one. lucb1e.com and another site are also not indexed, though I blocked it based on the user agent string. That way it doesn't get page data or non-HTML files from my server. I introduced this when they were pulling this AMP thing: https://lucb1e.com/?p=post&id=130 It personally doesn't impact me, but it impacts other people on the internet and I figured it was the only thing I can do…
You’d be better off just blocking all of the ip addresses that google crawls from. There are lists of those out there.
When I used to cloak website content, and only serve up certain content to google, the only reliable way was to use ip cloaking. Because google crawls using “partners”, such as using Comcast IPs.
So if you’re want to really get your site out of the index, serve up the page with a noindex tag or noindex in the server header based on google ip addresses.
Imagine wikipedia and the top newspapers doing this ... users will start to use another search engine.
Earlier quoted context omitted.
> I don't see why I would want to exclude my site from any index? Being in indexes helps people find my writing, which I like! It's essentially a form of boycott. If one believes Google is a problematic entity (too many fingers in too many aspects of our lives), it's a way to sever connections with them at some personal cost. At least, if you care about search traffic - one might argue the assumption that Google-like…
So, first, I don't think Google search is harmful, so I'm not especially interested in boycotting, but I'm happy to grant this for the discussion. There's already an interesting question about when boycotting is it worthwhile tactic, but in this situation, there's the additional complexity of there being two ends at which one could boycott a search engine: * Producer: don't allow the search engine to include your stu…
I appreciate that you mention in your bio that you do work in Ads at Google, which seems directly relevant to OP’s point about boycotting Google by blocking indexing. If Google can’t or otherwise doesn’t index your content, Google can’t profit from selling ads that it otherwise show alongside search results for it. If Google de-indexing became popular among a group of content creators, other search engines may not be similarly blocked, and other alternatives to find said content would be found or created, all to the detriment of Google Ads placement, which is a profitable - and inseparable - component of Google Search.
Oh hey I thought I was the only one. lucb1e.com and another site are also not indexed, though I blocked it based on the user agent string. That way it doesn't get page data or non-HTML files from my server. I introduced this when they were pulling this AMP thing: https://lucb1e.com/?p=post&id=130 It personally doesn't impact me, but it impacts other people on the internet and I figured it was the only thing I can do…
That’s a good idea,but google sometime crawls without the google user agent. So that’s not going to be 100 percent foolproof. You’d be better off just blocking all of the ip addresses that google crawls from. There are lists of those out there. When I used to cloak website content, and only serve up certain content to google, the only reliable way was to use ip cloaking. Because google crawls using “partners”, such a…
Is this different than when others use proxies to evade access controls.