Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nepenthes is a tarpit to catch AI web crawlers

zadzmo.org

Post reply on HN
Print view View original

141–150 of 290 posts

Re: Nepenthes is a tarpit to catch AI web crawlers

#141

Post by dzhiurgis »

dspillett wrote: post #62

Tarpits to slow down the crawling may stop them crawling your entire site, but they'll not care unless a great many sites do this. Your site will be assigned a thread or two at most and the rest of the crawling machine resources will be off scanning other sites. There will be timeouts to stop a particular site even keeping a couple of cheap threads busy for long. And anything like this may get you delisted from searc…

Can you put some topic in tarpit that you don't want LLMs to learn about? Say put bunch of info about competitor so that it learns to avoid it?

Re: Nepenthes is a tarpit to catch AI web crawlers

#142

Post by [deleted] »

hassleblad23 wrote: post #74
bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

I am not surprised that OpenAI is not interested if fixing this.

[deleted]

Re: Nepenthes is a tarpit to catch AI web crawlers

#143

Post by luckylion »

nerdix wrote: post #140

Are the big players (minus Google since no one blocks google bot) actively taking measures to circumvent things like Cloudflare bot protection? Bot detection is fairly sophisticated these days. No one bypasses it by accident. If they are getting around it then they are doing it intentionally (and probably dedicating a lot of resources to it). I'm pro-scraping when bots are well behaved but the circumvention of bot de…

Not that I've seen it.

If you enable Cloudflare Captcha, you'll see basically no more bots, only the most persistent remain (that have an active interest in you/your content and aren't just drive-by-hits).

It's just that having the brief interception hurts your conversion rate. Might depend on industry, but we saw 20-30% drops in page views and conversions which just makes it a nuclear option when you're under attack, but not something to use just to block annoyances.

Re: Nepenthes is a tarpit to catch AI web crawlers

#144

Post by larsrc »

dspillett wrote: post #62

Tarpits to slow down the crawling may stop them crawling your entire site, but they'll not care unless a great many sites do this. Your site will be assigned a thread or two at most and the rest of the crawling machine resources will be off scanning other sites. There will be timeouts to stop a particular site even keeping a couple of cheap threads busy for long. And anything like this may get you delisted from searc…

I've been considering setting up "ConfuseAIpedia" in a similar manner using sentence templates and a large set of filler words. Obviously with a warning for humans. I would set it up with an appropriate robots.txt blocking crawlers so only unethical crawlers would read it. I wouldn't try to tarpit beyond protecting my own server, as confusion rogue AI scrapers is more interesting than slowing them down a bit.

Re: Nepenthes is a tarpit to catch AI web crawlers

#145

Post by JTyQZSnP3cQGa8B »

GaggiX wrote: post #60

As always, I find it hilarious that some people believe that these companies will train their flagship model on uncurated data, and that text generated by a Markov chain will not be filtered out.

Then why the DDOS on random web sites?

Re: Nepenthes is a tarpit to catch AI web crawlers

#146

Post by WD-42 »

pmarreck wrote: post #132
quchen wrote: post #5

Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.

It's not. It's rather pointless and frankly, nearsighted. And we can DDoS sites like this just as offensively as well simply by making many requests to it since its own docs say its Markov generation is computationally expensive, but it is NOT expensive for even 1 person to make many requests to it. Just expensive to host. So feel free to use this bash function to defeat these: httpunch() { local url=$1 local connect…

You called the parent unintelligent yet need an LLM to show you how to run curl in a loop. Yikes.

Re: Nepenthes is a tarpit to catch AI web crawlers

#147

Post by tivert »

p0nce wrote: post #105
hartator wrote: post #12

There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.

Brand new site with no user gets 1k request a month by bots, the CO2 cost must be atrocious.

> Brand new site with no user gets 1k request a month by bots, the CO2 cost must be atrocious.

Yep: https://www.energy.gov/articles/doe-releases-new-report-eval...:

> The report finds that data centers consumed about 4.4% of total U.S. electricity in 2023 and are expected to consume approximately 6.7 to 12% of total U.S. electricity by 2028. The report indicates that total data center electricity usage climbed from 58 TWh in 2014 to 176 TWh in 2023 and estimates an increase between 325 to 580 TWh by 2028.

A graph in the report says in data centers used 1.9% in 2018.

Re: Nepenthes is a tarpit to catch AI web crawlers

#148

Post by GaggiX »

JTyQZSnP3cQGa8B wrote: post #145
GaggiX wrote: post #60

As always, I find it hilarious that some people believe that these companies will train their flagship model on uncurated data, and that text generated by a Markov chain will not be filtered out.

Then why the DDOS on random web sites?

I guess that depends on how the webspider is configured, I doubt the curation is done in real-time while scraping.

Re: Nepenthes is a tarpit to catch AI web crawlers

#149

Post by breakingcups »

jsheard wrote: post #33
taikahessu wrote: post #30

We had our non-profit website drained out of bandwidth and site closed temporarily (!!) from our hosting deal because of Amazon bot aggressively crawling like ?page=21454 ... etc. Gladly Siteground restored our site without any repercussions as it was not our fault. Added Amazon bot into robots.txt after that one. Don't like how things are right now. Is a tarpit the solution? Or better laws? Would they stop the chine…

For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site. https://github.com/ai-robots-txt/ai.robots.txt There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.

Such as OpenAI, who will ignore robots.txt and change their user agent to evade blocks, apparently[1]

1: https://www.reddit.com/r/selfhosted/comments/1i154h7/openai_...

Re: Nepenthes is a tarpit to catch AI web crawlers

#150

Post by [deleted] »

permo-w wrote: post #117
bflesch wrote: post #87

Earlier quoted context omitted.

Their security.txt email address replies and asks you to go on BugCrowd. BugCrowd staff is unwilling (or too incompetent) to run a bash curl command to reproduce the issue, while also refusing to forward it to OpenAI. The support@openai.com waits an hour before answering with ChatGPT answer. Issues raised on GitHub directly towards their engineers were not answered. Also Microsoft CERT & Azure security team do not re…

why try this hard for a private company that doesn't employ you?

[deleted]
Post reply on HN