Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Removing my site from Google search

btao.org

Post reply on HN
Print view View original

81–90 of 119 posts

Re: Removing my site from Google search

#81

Post by chihuahua »

AshamedCaptain wrote: post #18
TedDoesntTalk wrote: post #11

Earlier quoted context omitted.

Doesn’t that mean you have to know every IP Address used by Google bot now and in the future?

Not a very hard problem; after all, many websites allow full access to Googlebot IP ranges yet show a paywall to everyone else (including competing search engines). I also happen to ban Google ranges on multiple less-public sites specially since they completely ignore robots.txt and crawl-delay.

Is that how archive.vn works? I've always wondered how they are able to get the full text of paywalled sites like Wall Street Journal who give 0 free articles per month.

Re: Removing my site from Google search

#83

Post by jefftk »

aspenmayer wrote: post #78
jefftk wrote: post #75

Earlier quoted context omitted.

So, first, I don't think Google search is harmful, so I'm not especially interested in boycotting, but I'm happy to grant this for the discussion. There's already an interesting question about when boycotting is it worthwhile tactic, but in this situation, there's the additional complexity of there being two ends at which one could boycott a search engine: * Producer: don't allow the search engine to include your stu…

> (Disclosure: I work at Google, but not on Search, Chrome, or Gmail) I appreciate that you mention in your bio that you do work in Ads at Google, which seems directly relevant to OP’s point about boycotting Google by blocking indexing. If Google can’t or otherwise doesn’t index your content, Google can’t profit from selling ads that it otherwise show alongside search results for it. If Google de-indexing became popu…

I don't work on that kind of ad: I work in display ads. If you go to a newspaper or other publisher and see ads alongside the content, there's a good chance that my team owns the JS that handles requesting those ads and putting the responses on the screen.

Re: Removing my site from Google search

#84

Post by bhartzer »

bouke wrote: post #2

Why not use robots.txt instead of littering your html with googlebot instructions?

Google will still index the url even if you block them from crawling the page via robots.txt. They will index the lane, and it can still rank well. Google just puts up a message in the results saying they’re not allowed to crawl the page.

Re: Removing my site from Google search

#85

Post by Seirdy »

ignoramous wrote: post #62
DantesKite wrote: post #29

I like Google, but wouldn’t mind a better search engine, even at the cost of my privacy, so long as I had a choice for what could be shared.

You're in luck, since there's active development in this space: Neeva.com and kagi.com two of the many alt search engines.

I listed 25 alternative indexing search engines, including Neeva: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

I'd forgotten about Kagi, thanks; someone mentioned it to me on IRC but it slipped my mind. I'll add it to the list later today.

Re: Removing my site from Google search

#86

Post by qmarchi »

bhartzer wrote: post #76
lucb1e wrote: post #17

Oh hey I thought I was the only one. lucb1e.com and another site are also not indexed, though I blocked it based on the user agent string. That way it doesn't get page data or non-HTML files from my server. I introduced this when they were pulling this AMP thing: https://lucb1e.com/?p=post&id=130 It personally doesn't impact me, but it impacts other people on the internet and I figured it was the only thing I can do…

That’s a good idea,but google sometime crawls without the google user agent. So that’s not going to be 100 percent foolproof. You’d be better off just blocking all of the ip addresses that google crawls from. There are lists of those out there. When I used to cloak website content, and only serve up certain content to google, the only reliable way was to use ip cloaking. Because google crawls using “partners”, such a…

Hey! Googler here!

We don't use our hardware located on partner networks to do indexing. Those machines are searching for malware and serving some YouTube videos and Play Store downloads.

Re: Removing my site from Google search

#87

Post by forgotmypw17 »

markoutso wrote: post #71
forgotmypw17 wrote: post #68

I use simple HTTP auth with an easy username and password on most my sites. It is rarely a problem for anyone I invite, except perhaps Instagram's browser, but no crawler traffic.

That's a shitty solution. The whole point is to keep the website public.

I think it depends on who your audience is.

For "general" audience, I would use a proof-of-work puzzle 10 seconds long and a basic question captcha with human review.

But for a site which is primarily for tech-savvy people, with an accent on retro-compatibility (HTTP auth is supported almost universally, even by Mosaic), I can't think of a better option. Not that interested in SEO, since the software is my main target.

Re: Removing my site from Google search

#88

Post by PaulDavisThe1st »

pyrale wrote: post #61
cubano wrote: post #57

Earlier quoted context omitted.

Of course there are an infinite number of alternatives...I just quickly picked something that sounded decent that tried to make my point about my observation lately that Capitalism was getting knocked around everywhere I looked. In the last Econtalk podcast that I listened to last night, they discussed the loneliness "epidemic" and the author ended up blaming Thatcher-based Capitalism as perhaps the main reason why p…

> Capitalism [is] getting knocked around everywhere I looked. That's what happened with every social system in the past, and, while failures certainly have happened, we've always found ways for that criticism to result in improvements to our societies. It would be very surprising for the current shape of our society or, more generally, capitalism to be an exception to the rule, unless you subscribe to the "end of his…

No need to believe Fukiyama. The exception to the rule concept can be based on the idea that capitalism is novel in its ability to absorb and transform any protest against it. The divine right of kings, to pick just one example, was unable to pull off this trick, and perished under questioning. So far, capitalism has done remarkably well at incorporating protest and criticism that targets it, in a way that does seem quite novel.

Re: Removing my site from Google search

#89

Post by PaulDavisThe1st »

TFA cites a book on surveillance capitalism:

> …the essence of the exploitation here is the rendering of our lives as behavioral data for the sake of other’s improved control of us.

I don't doubt that surveillance capitalism is a problem. I do doubt that the underlying motive here on the part of capitalist concerns is improved control of us. Capitalism wants profit to move towards capital. Improved control might be a way of doing that, but it's only one tool in the arsenal of capital, and it's a fragile one at best.

It seems to me that the main motive of capitalism is to continue to drive high levels of desire-driven spending, and that it is at least as likely that surveillance capitalism is mostly about understanding how best to do that as it about actual control.

Yes, we know that marketing & advertising have demonstrated over the course of at least a century that consumer control is possible and desirable. It's just not clear that capitalism needs to increase or improve the level of control over what's already established.

Re: Removing my site from Google search

#90

Post by MichaelMoser123 »

PaulDavisThe1st wrote: post #89

TFA cites a book on surveillance capitalism: > …the essence of the exploitation here is the rendering of our lives as behavioral data for the sake of other’s improved control of us. I don't doubt that surveillance capitalism is a problem. I do doubt that the underlying motive here on the part of capitalist concerns is improved control of us. Capitalism wants profit to move towards capital. Improved control might be a…

Depends on how control is defined. Ultimately they want to steer your purchasing behavior, by showing you super focused ads on even more stuff that you would like to buy. That would be some kind of control.
Post reply on HN