Earlier quoted context omitted.
You called the parent unintelligent yet need an LLM to show you how to run curl in a loop. Yikes.
The 21st century script kiddy
Nepenthes is a tarpit to catch AI web crawlers
251–260 of 290 posts
Re: Nepenthes is a tarpit to catch AI web crawlers
#252I have a very vague concept for this, with a different implementation. Some, uh, sites (forums?) have content that the AI crawlers would like to consume, and, from what I have heard, the crawlers can irresponsibly hammer the traffic of said sites into oblivion. What if, for the sites which are paywalled, the signup, which invariably comes with a long click-through EULA, had a legal trap within it, forbidding ingestio…
Legal traps are not a thing.
Sure they are, they're called EULAs. What do you call clauses that force you to give up your right to sue another party in court other than a trap?
Re: Nepenthes is a tarpit to catch AI web crawlers
#253Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…
Am I correct in understanding that you waited at most one week for a reply?
In my experience with large companies, that's rather short. Some nudging may be required every now and then, but expecting a response so fast seems slightly unreasonable to me.
Re: Nepenthes is a tarpit to catch AI web crawlers
#254Earlier quoted context omitted.
How can it reach localhost or is this only a placeholder for a real address?
The code in the github repo has some errors to prevent script kiddies from directly copy/pasting it. Obviously the proof-of-concept shared with OpenAI/BugCrowd didn't have such errors.
Ah ok, thanks, that makes sense.
Btw the ChatGPT Web App (haven’t tested with the Desktop App) can find info from local/private sites with the search tool, i assume they browse with a client side function.
Re: Nepenthes is a tarpit to catch AI web crawlers
#255What blows my mind is that this is functionally a solved problem. The big search crawlers have been around for years & manage to mostly avoid nuking sites into oblivion. Then AI gang shows up - supposedly smartest guys around - and suddenly we're re-inventing the wheel on crawling and causing carnage in the process.
Search crawlers have the goal of directing people towards the websites they crawl. They have a symbiotic relationship, so they put in (some) effort not to blow websites out of the water with their crawling, because a website that's offline is useless for your search index. AI crawlers don't care about directing people towards websites. They intend to replace websites, and are only interested in copying whatever infor…
FWIW when I research stuff through chatgpt I click on the source links all the time. It usually only summarizes stuff. For ex: if you're shopping for a certain product it wont bring you to the store page where all the reviews are. It will just make a top ten list type thing quickly.
Re: Nepenthes is a tarpit to catch AI web crawlers
#256I'm actually quite happy with AI crawlers. I recently found out chatgpt suggest one of my sites when asked to suggest a good, independent site that covered the topic I searched for. Especially now that for instance chatgpt is adding source links, I think we should treat AI crawlers the same as search engine crawlers.
Re: Nepenthes is a tarpit to catch AI web crawlers
#257Unless this concept becomes a mass phenomenon with many implementations, isn’t this pretty easy to filter out? And furthermore, since this antagonizes billion-dollar companies that can spin up teams doing nothing but browse Github and HN for software like this to prevent polluting their datalakes, I wonder whether this is a very efficient approach.
Author of a similar tool here[0]. There are a few implementations of this sort of thing that I know of. Mine is different in that the primary purpose is to slightly alter content statically using a Markov generator, mainly to make it useless for content reposters, secondarily to make it useless to LLM crawlers that ignore my robots.txt file[1]. I assume the generated text is bad enough that the LLM crawlers just thro…
poison-the-wellms
I gotta give props for this project name.
Re: Nepenthes is a tarpit to catch AI web crawlers
#258Earlier quoted context omitted.
> first two words > "We the people" I don't know if that's a typo or intentional, but that's such a typical LLM thing to do. AI: where you make computers bad at the very basics of computing.
https://pressbooks.openedmb.ca/wordandsentencestructures/cha... I believe what the LLM replies with is in fact correct. From the standpoint of a programmer or any other category of people that are attuned to some kind of formal rigor? Absolutely not. But for any other kind of user who is more interested in the first two concepts instead, this is the thing to do.
No, I am quite sure that if you asked a random person on the street how many words are in “We the people”, they would say three.
Re: Nepenthes is a tarpit to catch AI web crawlers
#259The arms race between AI bots and bot-protection is only going to get worse, leading to increasing infra costs while negatively impacting the UX and performance (captchas, rate limiting, etc.). What's a reasonable way forward to deal with more bots than humans on the internet?
For me, this would finally be a good use case for bitcoin or similar digital transactions. Let the client provide either proof-of-work or proof-of-payment. If we can make the proof of work match the browsing speed of an average human, anything accessing more pages than that will need to provide payment instead.
Re: Nepenthes is a tarpit to catch AI web crawlers
#260Are the big players (minus Google since no one blocks google bot) actively taking measures to circumvent things like Cloudflare bot protection? Bot detection is fairly sophisticated these days. No one bypasses it by accident. If they are getting around it then they are doing it intentionally (and probably dedicating a lot of resources to it). I'm pro-scraping when bots are well behaved but the circumvention of bot de…
Not that I've seen it. If you enable Cloudflare Captcha, you'll see basically no more bots, only the most persistent remain (that have an active interest in you/your content and aren't just drive-by-hits). It's just that having the brief interception hurts your conversion rate. Might depend on industry, but we saw 20-30% drops in page views and conversions which just makes it a nuclear option when you're under attack…
we saw 20-30% drops in page views and conversions
Why do you attribute this to only the "brief interception"? Shouldn't the logical conclusion be that Cloudflare may block 20-30% of regular traffic?