Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nepenthes is a tarpit to catch AI web crawlers

zadzmo.org

Post reply on HN
Print view View original

161–170 of 290 posts

Re: Nepenthes is a tarpit to catch AI web crawlers

#161

Post by zcase »

jsheard wrote: post #33
taikahessu wrote: post #30

We had our non-profit website drained out of bandwidth and site closed temporarily (!!) from our hosting deal because of Amazon bot aggressively crawling like ?page=21454 ... etc. Gladly Siteground restored our site without any repercussions as it was not our fault. Added Amazon bot into robots.txt after that one. Don't like how things are right now. Is a tarpit the solution? Or better laws? Would they stop the chine…

For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site. https://github.com/ai-robots-txt/ai.robots.txt There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.

For those looking, this is the best I've found: https://blog.cloudflare.com/declaring-your-aindependence-blo...

Re: Nepenthes is a tarpit to catch AI web crawlers

#162

Post by angoragoats »

marginalia_nu wrote: post #157
angoragoats wrote: post #110

Earlier quoted context omitted.

This may be true for large, established crawlers for Google, Bing, et al. I don’t see how you can make this a blanket statement for all crawlers, and my own personal experience tells me this isn’t correct.

These things are so common having some way of dealing with them is basically mandatory if you plan on doing any sort of large scale crawling. That said, crawlers are fairly bug prone, so misbehaving crawlers is also a relatively common sight. It's genuinely difficult to properly test a crawler, and useless to build it from specs, since the realities of the web are so far off the charted territory, any test you build…

I am aware of all of the things you mention (I've built crawlers before).

My point was only that there are plenty of crawlers that don't operate in the way the parent post described. If you want to call them buggy that's fine.

Re: Nepenthes is a tarpit to catch AI web crawlers

#163

Post by TeMPOraL »

dawnerd wrote: post #113
jonatron wrote: post #91

Earlier quoted context omitted.

I just looked at the logs for a site, and I saw PerplexityBot is looking at the robots.txt and ignoring it. They don't provide a list of IPs to verify if it is actually them. Anyway, just for anyone with PerplexityBot in their user agent, they can get increasingly bad responses until the abuse stops.

Perplexity is exceptionally bad because they say they respect the robots.txt but clearly don't. When pressed on it they basically shrug and say too bad not put stuff in public if you don't want it crawled. They got a UA block in cloudflare and seems like that did the trick.

Interesting. Now they seem to claim that not only they follow robots.txt for crawling, but that they also broke under pressure and made the unfortunate decisions to have user requests follow robots.txt too.

https://www.perplexity.ai/de/hub/technical-faq/how-does-perp...

Re: Nepenthes is a tarpit to catch AI web crawlers

#166

Post by griomnib »

marc wrote: post #78

OpenAI doesn’t take security seriously. I reported a vulnerability to them that allowed you to get IP addresses of their paying customers. OpenAI responded “Not applicable” indicating they don’t think it was a serious issue. The PoC was very easy to understand and simple to replicate. Edit: I guess I might as well disclose it here since they don’t consider it an issue. They were/are(?) hot linking logo images of thir…

What do they take seriously?

Re: Nepenthes is a tarpit to catch AI web crawlers

#167

Post by griomnib »

0xf00ff00f wrote: post #93
kerkeslager wrote: post #72

Question: do these bots not respect robots.txt? I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites. The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an I…

> The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an IP visits that page, they get added to a blocklist which simply drops their connections without response for 24 hours. I love this idea!

Yeah, this is elegant as fuck.

Re: Nepenthes is a tarpit to catch AI web crawlers

#168

Post by hinkley »

dspillett wrote: post #62

Tarpits to slow down the crawling may stop them crawling your entire site, but they'll not care unless a great many sites do this. Your site will be assigned a thread or two at most and the rest of the crawling machine resources will be off scanning other sites. There will be timeouts to stop a particular site even keeping a couple of cheap threads busy for long. And anything like this may get you delisted from searc…

If it takes them 100 times the average crawl time to crawl my site, that is an opportunity cost to them. Of course 'time' is fuzzy here because it depends how they're batching. The way most bots work is to pull a fixed number of replies in parallel per target, so if you double your response time then you halve the number of request per hour they slam you with. That definitely affects your cluster size.

However if they split ask and answered, or other threads for other sites can use the same CPUs while you're dragging your feet returning a reply, then as you say, just IO delays won't slow them down. You've got to use their CPU time as well. That won't be accomplished by IO stalls on your end, but could potentially be done by adding some highly compressible gibberish on the sending side so that you create more work without proportionately increasing your bandwidth bill. But that's could be tough to do without increasing your CPU bill.

Re: Nepenthes is a tarpit to catch AI web crawlers

#169

Post by manquer »

permo-w wrote: post #117
bflesch wrote: post #87

Earlier quoted context omitted.

Their security.txt email address replies and asks you to go on BugCrowd. BugCrowd staff is unwilling (or too incompetent) to run a bash curl command to reproduce the issue, while also refusing to forward it to OpenAI. The support@openai.com waits an hour before answering with ChatGPT answer. Issues raised on GitHub directly towards their engineers were not answered. Also Microsoft CERT & Azure security team do not re…

why try this hard for a private company that doesn't employ you?

While others (and OP) give good reasons, beyond passion and interest, those I see are typically doing this without a bounty to a build public profile to establish reputation that helps with employment or building their devopssec consulting practices.

Unlike clear cut security issues like RCEs, (D)DoS and social engineering few other classes of issues are hard to process for devopssec, it is a matter of product design, beyond the control of engineering.

Say for example if you offer but do not require 2FA usage to users, having access to known passwords for some usernames from other leaks then with a rainbow table you can exploit poorly locked down accounts.

Similarly many dev tools and data stores for ease of adoption of their cloud offerings may be open by default, i.e. no authentication, publicly available or are easy to misconfigure poorly that even a simple scan on shodan would show. On a philosophical level these security issues in product design perhaps, but no company would accept those as security vulnerabilities, thankfully this type of issues is reducing these days.

When your inbox starts filling up with reporting items like this to improve their cred, you stop engaging because the product teams will not accept it and you cannot do anything about it, sooner or later devopsec teams tend to outsource initial filtering to bug bounty programs and they obviously do not a great job of responding especially when it is one of the grayer categories.

Re: Nepenthes is a tarpit to catch AI web crawlers

#170

Post by slongfield »

NathanKP wrote: post #81

This looks extremely easy to detect and filter out. For example: https://i.imgur.com/hpMrLFT.png In short, if the creator of this thinks that it will actually trick AI web crawlers, in reality it would take about 5 mins of time to write a simple check that filters out and bans the site from crawling. With modern LLM workflows its actually fairly simple and cheap to burn just a little bit of GPU time to check if the d…

The most annoying bots are the ones that mindlessly slam sites over and over, without doing any filtering. Having these kinds of tarpits out in the wild forcing people to be better behaved with their crawling bots is a feature, not a bug.
Post reply on HN