Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nepenthes is a tarpit to catch AI web crawlers

zadzmo.org

Post reply on HN
Print view View original

181–190 of 290 posts

Re: Nepenthes is a tarpit to catch AI web crawlers

#184

Post by upwardbound2 »

upwardbound2 wrote: post #182

Is Nepenthes being mirrored in enough places to keep the community going if the original author gets any DMCA trouble or anything? I'd be happy to host a mirror but am pretty busy and I don't want to miss a critical file by accident.

It looks like someone saved a copy of the downloads page and the three linked files in the wayback machine yesterday, so that's good at least. https://web.archive.org/web/20250000000000*/https://zadzmo.o...

Re: Nepenthes is a tarpit to catch AI web crawlers

#185

Post by RamblingCTO »

Why wouldn't a max-depth (which I always implement in my crawlers if I write any) prevent any issues you'd have? Am I overlooking something? Or does it run under the assumption that the crawlers they are targeting are so greedy that they don't have max-depth/a max number of pages for a domain?

Re: Nepenthes is a tarpit to catch AI web crawlers

#186

Post by maeil »

zcase wrote: post #161
jsheard wrote: post #33

Earlier quoted context omitted.

For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site. https://github.com/ai-robots-txt/ai.robots.txt There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.

For those looking, this is the best I've found: https://blog.cloudflare.com/declaring-your-aindependence-blo...

This seemed to work for some time when it came out but IME no longer does.

Re: Nepenthes is a tarpit to catch AI web crawlers

#187

Post by johnisgood »

gruez wrote: post #180
johnisgood wrote: post #172

Earlier quoted context omitted.

How is that infinite if the last one is always the same? Am I misunderstanding this? I assumed it is almost like an infinite scroll or something.

Here's another site that does something similar (iterating over bitcoin private keys rather than uuids), but has separate pages and would theoretically catch a crawler: https://allprivatekeys.com/all-bitcoin-private-keys-list

503 :D

Re: Nepenthes is a tarpit to catch AI web crawlers

#188

Post by krior »

NathanKP wrote: post #152
canu7 wrote: post #129

Earlier quoted context omitted.

If they need to query a trained LLM for each page they crawl, I would guess that the training cost would scale up pretty badly...

Of course you wouldn't do it for every single page. If I was designing this crawler I'd make it sample a percentage of pages, starting at 100% sample rate for a completely unknown website, decreasing the sample rate over time as more "good" pages are found relative to "bad" pages. After a "good" page percentage threshold is exceeded, stop sampling entirely and just crawl, assuming that all content is good. After a "b…

> After a "bad" page percentage threshold is exceeded just stop wasting your time crawling that domain entirely.

In the words of Bush jr.: Mission accomplished!

Re: Nepenthes is a tarpit to catch AI web crawlers

#189

Post by khana »

bflesch wrote: post #87
hassleblad23 wrote: post #74

Earlier quoted context omitted.

I am not surprised that OpenAI is not interested if fixing this.

Their security.txt email address replies and asks you to go on BugCrowd. BugCrowd staff is unwilling (or too incompetent) to run a bash curl command to reproduce the issue, while also refusing to forward it to OpenAI. The support@openai.com waits an hour before answering with ChatGPT answer. Issues raised on GitHub directly towards their engineers were not answered. Also Microsoft CERT & Azure security team do not re…

[dead]
Post reply on HN