Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nepenthes is a tarpit to catch AI web crawlers

zadzmo.org

Post reply on HN
Print view View original

271–280 of 290 posts

Re: Nepenthes is a tarpit to catch AI web crawlers

#271

Post by alt187 »

pmarreck wrote: post #132
quchen wrote: post #5

Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.

It's not. It's rather pointless and frankly, nearsighted. And we can DDoS sites like this just as offensively as well simply by making many requests to it since its own docs say its Markov generation is computationally expensive, but it is NOT expensive for even 1 person to make many requests to it. Just expensive to host. So feel free to use this bash function to defeat these: httpunch() { local url=$1 local connect…

The tarpit is made for LLM crawlers who don't respect robots.txt. Do you love LLMs so much that you wish that they wouldn't have to respect this stupid, anticorporate AI-doomer robots.txt convention so they can pry out of the greedy hands of the webserver one more URL?

Maybe you just had a knee-jerk reaction.

Re: Nepenthes is a tarpit to catch AI web crawlers

#272

Post by WD-42 »

pmarreck wrote: post #249
WD-42 wrote: post #146

Earlier quoted context omitted.

You called the parent unintelligent yet need an LLM to show you how to run curl in a loop. Yikes.

Your assumption that I couldn't have written this myself or that I didn't make corrections to it is telling. I've only been doing dev for 30+ years lol LLMs are an accelerant, like all previous tools... Not a replacement, although it seems most people still need to figure that out for themselves while I already have

I see you edited out the part where you made fun of the author using Lua and then called them stupid. It’s a bad look, especially using all that space to paste a script that could be reduced to 5 lines, which is pretty typical of the slop that the author has an issue with in the first place.

Re: Nepenthes is a tarpit to catch AI web crawlers

#273

Post by bflesch »

mitjam wrote: post #254
bflesch wrote: post #192

Earlier quoted context omitted.

The code in the github repo has some errors to prevent script kiddies from directly copy/pasting it. Obviously the proof-of-concept shared with OpenAI/BugCrowd didn't have such errors.

Ah ok, thanks, that makes sense. Btw the ChatGPT Web App (haven’t tested with the Desktop App) can find info from local/private sites with the search tool, i assume they browse with a client side function.

Yeah I first wanted to use this bug to scan their IP ranges and figure out their internal network (e.g. make requests to 10.0.0.1, 10.0.0.2, and so on). But then I realized that it will hallucinate an answer for every IP it is given :)

So it would just come up with titles of random router admin panel websites.

Re: Nepenthes is a tarpit to catch AI web crawlers

#274

Post by marginalia_nu »

Dwedit wrote: post #118
dawnerd wrote: post #113

Earlier quoted context omitted.

Perplexity is exceptionally bad because they say they respect the robots.txt but clearly don't. When pressed on it they basically shrug and say too bad not put stuff in public if you don't want it crawled. They got a UA block in cloudflare and seems like that did the trick.

User Agent block just means they'd spoof their user agent.

That generally gives you even more trouble with cloudflare. Behaving in any way inconsistent with your UA string is one of the easiest methods of identifying bots.

Yeah you can use headless browsers, but then you're also using orders of magnitude more compute, and that's not really something that scales.

The best way to avoid ending up in captcha-land is to say who you are, and respect robots.txt.

Re: Nepenthes is a tarpit to catch AI web crawlers

#275

Post by manquer »

bflesch wrote: post #234
manquer wrote: post #169

Earlier quoted context omitted.

While others (and OP) give good reasons, beyond passion and interest, those I see are typically doing this without a bounty to a build public profile to establish reputation that helps with employment or building their devopssec consulting practices. Unlike clear cut security issues like RCEs, (D)DoS and social engineering few other classes of issues are hard to process for devopssec, it is a matter of product design…

I've been on the receiving end of many low-effort vulnerability reports so I have sympathy for people who would feel that way. However this was reported under my clear name, my credentials are visible online, and it was a ready-to-execute proof-of-concept. Speculation: I'm convinced that this API endpoint was one of their "AI agents" because you could also send ChatGPT commands via the `urls[]` parameter and it was a…

Yes I understand, what you describe is something I would definitely consider as a security issue.

However just like how say the DoS using SYN floods was not treated as an important issue by ISPs and other network operators for a long time, I am not surprised OpenAI/Microsoft is treating yours not seriously.

The attitude typically would as long as it doesn't affect my services it is not my job to worry about it, until it becomes a PR issue.

Re: Nepenthes is a tarpit to catch AI web crawlers

#276

Post by bflesch »

Xmd5a wrote: post #262
kevinventullo wrote: post #258

Earlier quoted context omitted.

No, I am quite sure that if you asked a random person on the street how many words are in “We the people”, they would say three.

Indeed, but consider this situation: You have a collection of documents and want to extract the first n words because you're interested in the semantic content of the beginning of each doc. You use a LLM because why not. The LLM processes the documents, and every now and then it returns a slightly longer or shorter list of words because it better captures the semantic content. I'd argue the LLM is in fact doing exact…

Good explanation. That's most likely the reason for it.

At the same time it's what I don't like with most modern search functions: they won't allow you to search for exact words or sentences. It doesn't work on google, last time I played around with elasticsearch it didn't work, and it happens in many other places.

Obviously if you want performance you need to group common words and ignore punctuation. But if you're doing code search for actual strings (like on github) it's a totally different problem.

Would be nice to have a google-like search index that you can query with regexp.

Re: Nepenthes is a tarpit to catch AI web crawlers

#277

Post by luckylion »

tremon wrote: post #260
luckylion wrote: post #143

Earlier quoted context omitted.

Not that I've seen it. If you enable Cloudflare Captcha, you'll see basically no more bots, only the most persistent remain (that have an active interest in you/your content and aren't just drive-by-hits). It's just that having the brief interception hurts your conversion rate. Might depend on industry, but we saw 20-30% drops in page views and conversions which just makes it a nuclear option when you're under attack…

we saw 20-30% drops in page views and conversions Why do you attribute this to only the "brief interception"? Shouldn't the logical conclusion be that Cloudflare may block 20-30% of regular traffic?

There was no hard block, just everyone getting the brief JS-captcha-thing.

The reason I'm fairly confident that it wasn't bots that were blocked is that we run quite a bit of bot-filtering in our analysis (and only track via delayed JS, which already gets rid of of 80-90% of bots; we don't care about bots viewing our pages, but we don't want them to mess up our stats) and all other metrics (conversion-rate, device + browser-distribution, country-mix) still lined up, they were just significantly lower than what expectations + search analytics said they should be.

Re: Nepenthes is a tarpit to catch AI web crawlers

#278

Post by nurettin »

bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

They don't care. You are just raising their costs which they will in return charge their customers.

Re: Nepenthes is a tarpit to catch AI web crawlers

#279

Post by kerkeslager »

jonatron wrote: post #83
kerkeslager wrote: post #72

Question: do these bots not respect robots.txt? I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites. The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an I…

You haven't seen any problems because you created a solution to the problem!

Well, I wasn't the original developer who set up every site I work on. Some of the sites I work on don't have this implemented because I wasn't the one who set them up initially.

Re: Nepenthes is a tarpit to catch AI web crawlers

#280

Post by Dwedit »

kerkeslager wrote: post #72

Question: do these bots not respect robots.txt? I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites. The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an I…

Even something like a special URL that auto-bans you can be abused by pranksters. Simply embedding an tag that fetches the offending URL could trigger it, as well as tricking people into clicking a link.
Post reply on HN