Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nepenthes is a tarpit to catch AI web crawlers

zadzmo.org

Post reply on HN
Print view View original

281–290 of 290 posts

Re: Nepenthes is a tarpit to catch AI web crawlers

#281

Post by kerkeslager »

Dwedit wrote: post #280
kerkeslager wrote: post #72

Question: do these bots not respect robots.txt? I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites. The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an I…

Even something like a special URL that auto-bans you can be abused by pranksters. Simply embedding an tag that fetches the offending URL could trigger it, as well as tricking people into clicking a link.

Ehhh, is there any reason I should be worried about that? The tag would have to be in a spot where users are likely to go, otherwise users will never view the tag. A link of any kind to the honeypot isn't likely to, for example, go viral on social media, because it's going to appear as a broken link/image and nobody will upvote it. I'm not seeing an attack vector that gets this link in front of my users with enough frequency to be worth considering.

A bigger concern is arguably users who are all behind the same IP address, i.e. some of the sites I work on have employee-only parts which can only be accessed via VPN, so in theory one employee could get the whole company banned, and that would be tricky to figure out. So far that hasn't been a problem, but now that I'm thinking about it, maybe I should have a whitelist override for that. :)

Re: Nepenthes is a tarpit to catch AI web crawlers

#282

Post by jesprenj »

Dwedit wrote: post #280
kerkeslager wrote: post #72

Question: do these bots not respect robots.txt? I haven't added these scrapers to my robots.txt on the sites I work on yet because I haven't seen any problems. I would run something like this on my own websites, but I can't see selling my clients on running this on their websites. The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an I…

Even something like a special URL that auto-bans you can be abused by pranksters. Simply embedding an tag that fetches the offending URL could trigger it, as well as tricking people into clicking a link.

This could be mitigited by having a special secret token in this honeypot URL that limits the time validity of the honeypot url and limits the IP address that this URL is for, let's say: hhtp://example/honeypot/hex(sha256(ipaddress | today(yyyy-mm-dd) | secret))

This special URL with the token would be in an anchor tag somewhere in the footer of every website, but hidden by a CSS rule and "Disallow: /honeypot" rule would be included in robots.txt.

Re: Nepenthes is a tarpit to catch AI web crawlers

#283

Post by p0nce »

p0nce wrote: post #105
hartator wrote: post #12

There are already “infinite” websites like these on the Internet. Crawlers (both AI and regular search) have a set number of pages they want to crawl per domain. This number is usually determined by the popularity of the domain. Unknown websites will get very few crawls per day whereas popular sites millions. Source: I am the CEO of SerpApi.

Brand new site with no user gets 1k request a month by bots, the CO2 cost must be atrocious.

s/a month/a day

Re: Nepenthes is a tarpit to catch AI web crawlers

#284

Post by bwfan123 »

ezrast wrote: post #130
bwfan123 wrote: post #50

Earlier quoted context omitted.

indeed. this will spur research on how to distinguish BS from legit content. which is the fundamental hallucination problem in llms. and all of us will benefit from this.

You can't programatically detect novel BS any more than you can programatically detect viruses or spam. You can only add the fingerprints of known badness into an ever-growing database. Viruses and spam are antagonistic to well-resourced institutions, and their databases get maintained reasonably well. LLM slop is being generated by those same well-resourced institutions. I don't think it fits into the same category…

the extent to which a random string is a "theorem" of some system can be measured i am guessing.

ie, given some text, to what extent is it "grounded" in some sense in facts.

Re: Nepenthes is a tarpit to catch AI web crawlers

#285

Post by netdevphoenix »

bflesch wrote: post #236
netdevphoenix wrote: post #227

Earlier quoted context omitted.

That's really bad. But then again OpenAI was he coolest company for a year two and now it's facing multiple existential crises. Chances are that the company won't be around by 2030 or will be partially absorbed by Microsoft. My take is that GPT-5 will never come out if it ever does it will just be to mark the official downfall of the company because it will fail to live to the expectations and will drop the valuation…

I've reframed ChatGPT as a google alternative without ads and am really happy when using it this way. It's still a great product and they'll be able to monetize it with ads just like google did. Personally it's quite disappointing because I'd have expected at least some engineer to say "it's not a bug it's a feature" or "thanks for informative vulnerability report, we'll fix it in next release". But just ignoring it…

Sadly, it doesn't work that way for OpenAI. OpenAI exists because it promised to be THE AI Company not just another LLM company. Its valuation reflects that. The promises made by the company reflects that. I think OpenAI is basically the guinea pig of this whole LLM experiment. Anything that OpenAI achieves, their competitors achieve in less time, with less resources and investment. And when the day comes that OpenAI falls, the money lost will be much greater than that of the other LLM companies

Re: Nepenthes is a tarpit to catch AI web crawlers

#286

Post by anthony42c »

bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

Where does the 5000 HTTP request limit come from? Is that the limit of the URLs array?

I was curious to learn more about the endpoint, but can't find any online API docs. The docs ChatGPT suggests are defined for api.openapi.com, rather than chatgpt.com/backend-api.

I wonder if its reasonable (from a functional perspective) for the attributions endpoint not to place a limit on the number of urls used for attribution. I guess potentially ChatGPT could reference hundreds of sites and thousands of web pages in searching for a complex question that covered a range of different interrelated topics? Or do I misunderstand the intended usage of that endpoint?

Re: Nepenthes is a tarpit to catch AI web crawlers

#287

Post by [deleted] »

bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

[deleted]

Re: Nepenthes is a tarpit to catch AI web crawlers

#288

Post by pabs3 »

bflesch wrote: post #54

Haha, this would be an amazing way to test the ChatGPT crawler reflective DDOS vulnerability [1] I published last week. Basically a single HTTP Request to ChatGPT API can trigger 5000 HTTP requests by ChatGPT crawler to a website. The vulnerability is/was thoroughly ignored by OpenAI/Microsoft/BugCrowd but I really wonder what would happen when ChatGPT crawler interacts with this tarpit several times per second. As C…

Could those 5000 HTTP requests be made to go back to the ChatGPT API?

Re: Nepenthes is a tarpit to catch AI web crawlers

#289

Post by jjuhl »

Why just catch the ones ignoring robots.txt? Why not explicitly allow them to crawl everything, but silently detect AI bots and quietly corrupt the real content so it becomes garbage to them while leaving it unaltered for real humans? Seems to me that would have a greater chance of actually poisoning their models and eventually make this AI/LLM crap go away.
Post reply on HN