Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Nginx to block Meta, Twitter and ChatGPT access to your sites

gist.github.com

Post reply on HN
Print view View original

21–26 of 26 posts

Re: Using Nginx to block Meta, Twitter and ChatGPT access to your sites

#21

Post by m3047 »

The most permanent and effective solutions (in terms of minimizing adversarial activity over time and destroying the value of what is harvested) involve serving fake content (poison!), making site failures sporadic (forcing them to maintain state), and making some of those errors look like they're upstream not something you're doing on a specific machine (really bad luck mate!).

The deadenders who felt it was worth it will keep trying for at least a while; the new exploiters will tend to give up sooner. robots.txt is a courtesy. Not everybody puts stuff on the internet with a working theory that your experience is more important than theirs.

Re: Using Nginx to block Meta, Twitter and ChatGPT access to your sites

#22

Post by eindiran »

tenebrisalietum wrote: post #19
IceWreck wrote: post #18

Earlier quoted context omitted.

And you think Meta or Twitter's bots would be dumb ? If they wanted to scrape your site, nobody can prevent them.

Kinda. Meta and Twitter want you to join their platforms, they aren't general purpose search engines scraping the entire Internet - they're scraping people that join them. Requests from Meta/Twitter are probably from a link someone put in a post. ChatGPT can't be an impolite Internet citizen (spoofing UA's) and claim to be using AI for the good for humanity, so they're not going to be dishonest with their user-agent.

> ChatGPT can't be an impolite Internet citizen (spoofing UA's) and claim to be using AI for the good for humanity, so they're not going to be dishonest with their user-agent.

That reads an awful-lot like "Google can't be evil and claim that their motto is 'Don't be evil', so they're not going to be evil" but here we are. The profit motive eventually undoes any principled claim by a company.

Re: Using Nginx to block Meta, Twitter and ChatGPT access to your sites

#24

Post by altairprime »

kisamoto wrote: post #10

Copying my comment here for additional discussion: Worth noting that most of these bots are 'good bots' (i.e. they will obey robots.txt). So you can avoid the nginx resource usage entirely by adding suitable robots.txt entries. I think using nginx tests like this could have negative effects on showing OpenGraph metadata (including images). If choosing this approach however I would probably respond with a 403 code to…

I advise 402 Payment Required \n Location: mailto:payment-offers@yourdomain. It’s still 4xx so they’ll probably know not to hit it repeatedly, and if they reach out to negotiate payment, you have a choice whether to accept or not.

Re: Using Nginx to block Meta, Twitter and ChatGPT access to your sites

#25

Post by danradunchev »

For everyone who miss the point.

- I don't want my content on those sites in any form and I don't want my content to feed their algorithms. So I do not care for opengraph or previews.

- Using robot.txt assumes they will 'obey' it. But they may choose not to. Its not mandatory in anyway.

- Yes, they can fake UA. This does not mean I should not take any measures to block them just becasue they can fake.

Re: Using Nginx to block Meta, Twitter and ChatGPT access to your sites

#26

Post by tenebrisalietum »

eindiran wrote: post #22
tenebrisalietum wrote: post #19

Earlier quoted context omitted.

Kinda. Meta and Twitter want you to join their platforms, they aren't general purpose search engines scraping the entire Internet - they're scraping people that join them. Requests from Meta/Twitter are probably from a link someone put in a post. ChatGPT can't be an impolite Internet citizen (spoofing UA's) and claim to be using AI for the good for humanity, so they're not going to be dishonest with their user-agent.

> ChatGPT can't be an impolite Internet citizen (spoofing UA's) and claim to be using AI for the good for humanity, so they're not going to be dishonest with their user-agent. That reads an awful-lot like "Google can't be evil and claim that their motto is 'Don't be evil', so they're not going to be evil" but here we are. The profit motive eventually undoes any principled claim by a company.

Absolutely, but until then, adding bot UA's to a blocklist is somewhat useful.

Like anything else in IT security it's never "set and forget" permanently; the effectiveness of things like that decay over time and must be periodically re-evaluated.

But if something can be used to your advantage now, even if for a while, then why not use it.

Post reply on HN