Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nepenthes is a tarpit to catch AI web crawlers

zadzmo.org

Post reply on HN
Print view View original

211–220 of 290 posts

Re: Nepenthes is a tarpit to catch AI web crawlers

#211

Post by marginalia_nu »

Havoc wrote: post #203

What blows my mind is that this is functionally a solved problem. The big search crawlers have been around for years & manage to mostly avoid nuking sites into oblivion. Then AI gang shows up - supposedly smartest guys around - and suddenly we're re-inventing the wheel on crawling and causing carnage in the process.

I think it's largely the mindset of moving fast and breaking things that's at fault. If say ship it at "good enough", it will not behave well.

Building a competent well-behaved crawler is a big effort that requires relatively deep understanding of more or less all web tech, and figuring out a bunch of stuff that is not documented anywhere and not part of any specs.

Re: Nepenthes is a tarpit to catch AI web crawlers

#212

Post by freetonik »

jeroenhd wrote: post #204
Havoc wrote: post #203

What blows my mind is that this is functionally a solved problem. The big search crawlers have been around for years & manage to mostly avoid nuking sites into oblivion. Then AI gang shows up - supposedly smartest guys around - and suddenly we're re-inventing the wheel on crawling and causing carnage in the process.

Search crawlers have the goal of directing people towards the websites they crawl. They have a symbiotic relationship, so they put in (some) effort not to blow websites out of the water with their crawling, because a website that's offline is useless for your search index. AI crawlers don't care about directing people towards websites. They intend to replace websites, and are only interested in copying whatever infor…

>Search crawlers have the goal of directing people towards the websites they crawl. They have a symbiotic relationship, so they put in (some) effort not to blow websites out of the water with their crawling, because a website that's offline is useless for your search index.

Ultimately not true. Google started showing pre-parsed "quick cards" instead of links a long time ago. The incentives of ad-driven search engines are to keep the visitors on the search engine rather than direct them to the source.

Re: Nepenthes is a tarpit to catch AI web crawlers

#213

Post by thruway516 »

WD-42 wrote: post #146
pmarreck wrote: post #132

Earlier quoted context omitted.

It's not. It's rather pointless and frankly, nearsighted. And we can DDoS sites like this just as offensively as well simply by making many requests to it since its own docs say its Markov generation is computationally expensive, but it is NOT expensive for even 1 person to make many requests to it. Just expensive to host. So feel free to use this bash function to defeat these: httpunch() { local url=$1 local connect…

You called the parent unintelligent yet need an LLM to show you how to run curl in a loop. Yikes.

The 21st century script kiddy

Re: Nepenthes is a tarpit to catch AI web crawlers

#214

Post by pona-a »

It feels like a Markov chain isn't adversarial enough.

Maybe you can use an open-weights model, assuming that all LLMs converge on similar representations, and use beam-search with inverted probability and repetition penalty or just GPT-2/LLaMA outwith with amplified activations to try and bork the projection matrices, return write pages and pages of phonetically faux English text to affect how the BPE tokenizer gets fitted, or anything else more sophisticated and deliberate than random noise.

All of these would take more resources than a Markov chain, but if the scraper is smart about ignoring such link traps, a periodically rotated selection of adversarial examples might be even better.

Nightshade had comparatively great success, discounting that its perturbations aren't that robust to rescaling. LLM training corpora are filtered very coarsely and take all they can get, unlike the more motivated attacker in Nightshade's threat model trying to fine-tune on one's style. Text is also quite hard to alter without a human noticing, except annoying zero-width Unicode which is easily stripped, so there's no presence of preserving legibility; I think it might work very well if seriously attempted.

Re: Nepenthes is a tarpit to catch AI web crawlers

#215

Post by jackcviers3 »

fc417fc802 wrote: post #107
bflesch wrote: post #99

Earlier quoted context omitted.

The technical flaws are quite trivial to spot, if you have the relevant experience: - urls[] parameter has no size limit - urls[] parameter is not deduplicated (but their cache is deduplicating, so this security control was there at some point but is ineffective now) - their requests to same website / DNS / victim IP address rotate through all available Azure IPs, which gives them risk of being blocked by other hoste…

> rotate through all available Azure IPs, ... They should come from the same IP address. I would guess that this is intentional, intended to prevent IP level blocks from being effective. That way blocking them means blocking all of Azure. Too much collateral damage to be worth it.

It is. There are scraping third party services you can pay for that will do all of this for you, and getting blocked by IP. You then make your request to the third-party scraper, receive the contents, and do with them whatever you need to do.

Re: Nepenthes is a tarpit to catch AI web crawlers

#216

Post by aleph_minus_one »

netdevphoenix wrote: post #208
BoredPositron wrote: post #206

Earlier quoted context omitted.

You are on hackernews. It’s curiosity not only about the flaw in their system but also how they as a system react to the flaw. Tells you a lot about companies you can later avoid when recruiters knock or you send out resumes.

I know I am on HN. Curiosity is one thing, investigating issues for free for a rich company is another. The former makes sense to me. The latter not as much, when we live in a world with all sorts of problems that are available to be solved. I think judging the future state of a company based on its present state is not really fair or reliable especially as the period between the two states gets wider. Culture change…

> I know I am on HN. Curiosity is one thing, investigating issues for free for a rich company is another.

The vulnerability https://github.com/bf/security-advisories/blob/main/2025-01-... targets other sites than OpenAI. OpenAI's crawler is rather the instrument of the crime for the attack.

Since this "just" leads to a potential reputation damage for OpenAI (and OpenAI's reputation is by now bad), and the victims are operators of other websites, I can see why OpenAI sees no urgency for fixing this bug.

Re: Nepenthes is a tarpit to catch AI web crawlers

#217

Post by netdevphoenix »

aleph_minus_one wrote: post #216
netdevphoenix wrote: post #208

Earlier quoted context omitted.

I know I am on HN. Curiosity is one thing, investigating issues for free for a rich company is another. The former makes sense to me. The latter not as much, when we live in a world with all sorts of problems that are available to be solved. I think judging the future state of a company based on its present state is not really fair or reliable especially as the period between the two states gets wider. Culture change…

> I know I am on HN. Curiosity is one thing, investigating issues for free for a rich company is another. The vulnerability https://github.com/bf/security-advisories/blob/main/2025-01-... targets other sites than OpenAI. OpenAI's crawler is rather the instrument of the crime for the attack. Since this "just" leads to a potential reputation damage for OpenAI (and OpenAI's reputation is by now bad), and the victims are…

I get it now. Thanks for the input

Re: Nepenthes is a tarpit to catch AI web crawlers

#219

Post by dspillett »

hinkley wrote: post #168
dspillett wrote: post #62

Tarpits to slow down the crawling may stop them crawling your entire site, but they'll not care unless a great many sites do this. Your site will be assigned a thread or two at most and the rest of the crawling machine resources will be off scanning other sites. There will be timeouts to stop a particular site even keeping a couple of cheap threads busy for long. And anything like this may get you delisted from searc…

If it takes them 100 times the average crawl time to crawl my site, that is an opportunity cost to them. Of course 'time' is fuzzy here because it depends how they're batching. The way most bots work is to pull a fixed number of replies in parallel per target, so if you double your response time then you halve the number of request per hour they slam you with. That definitely affects your cluster size. However if the…

> If it takes them 100 times the average crawl time to crawl my site, that is an opportunity cost to them.

If it takes 100 times the average crawl time per page on your site, which is one of many tens (hundreds?) of thousand sites, many of which may be bigger, unless they are doing one site at a time, so your site causes a full queue stall, such efforts likely amount to no more than statistical noise.

Post reply on HN