Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

311–320 of 691 posts

Re: Creepy Crawlies

#311

Post by worldsavior »

embedding-shape wrote: post #179
robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

> I have an admin dashboard with a little leaderboard for which ones get the most stuffed You can't tease us like that without showing screenshots.

NGL, that's something a crawler owner would say.

Re: Creepy Crawlies

#312

Post by nubinetwork »

inigyou wrote: post #290
Symbiote wrote: post #102

Earlier quoted context omitted.

Meta is an exception. They intentionally ignore robots.txt rules. (No second chances, I blocked their whole IP range.)

Everyone ignores robots.txt because every robots.txt blocks everyone.

I allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.

Re: Creepy Crawlies

#313

Post by wingworks »

inigyou wrote: post #281
Velocifyer wrote: post #115

I would add cloudflare, but set it to cache only mode *without* the bot blocking features.

No point, they are all unique requests.

I think he means, get cloudflare to cache your content, so the traffic never reaches your servers to begin with. Assuming your sites content is cacheable by cloudflare. I agree it's a sad state of affairs if you have to rely on a 3rd party..

Maybe I'm not understanding how many requests at a time bots are sending to kernel.org (or how larger kernel is), but couldn't they have a local cache system too, where all it has to do it serve up dumb .html pages, needing next to no compute cycles.

Re: Creepy Crawlies

#314

Post by Skunkleton »

Kuinox wrote: post #190

1.4 billions requests, 258 160 cpu hours. That's 1.5 requests per second ? I'm starting to believe, the issue is more that their software is not well optimized.

You are forgetting that there are multiple multi-core servers.

Re: Creepy Crawlies

#315

Post by apitman »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

This comment captures the current AI zeitgeist about as well as anything I've seen.

Re: Creepy Crawlies

#316

Post by LatticeAnimal »

inigyou wrote: post #198
LatticeAnimal wrote: post #152

Earlier quoted context omitted.

Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently. I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...

You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses.

Cool, so then that invalidates the ASIC problem, right?

My earlier idea was to imagine that each day Anubis picks an entirely different problem-class. Ex: one day it is Sha256, the next it is prime factorization, the next it is twin-prime-finding, the next it is cracking elliptic curves, the next it is some kind of sorting / information theory problem...

All with the goal of adapting constantly so that scrapers have a harder time optimizing for the PoW problem (i.e. with Sha256 ASICs)

Re: Creepy Crawlies

#317

Post by Kuinox »

Skunkleton wrote: post #314
Kuinox wrote: post #190

1.4 billions requests, 258 160 cpu hours. That's 1.5 requests per second ? I'm starting to believe, the issue is more that their software is not well optimized.

You are forgetting that there are multiple multi-core servers.

It's incredibly slow for a single core, that's my reference point.

Re: Creepy Crawlies

#318

Post by JsonCameron »

rplnt wrote: post #262
Aurornis wrote: post #187

Earlier quoted context omitted.

> Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time. So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views. A user who needs to load a single page and…

Now you have a session of sorts and can limit the requests for that client, right? They can be fast, just limited in volume - regular user isn't punished.

yep, that's the exact play. Or better fingerprinted & blocked in other ways

Re: Creepy Crawlies

#319

Post by bigstrat2003 »

kalkin wrote: post #80
lkbm wrote: post #15

> So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done. If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has…

This tick about "if AI smart how come crawler dumb" is in most complaints I've read about AI crawlers and I've started to find it pretty annoying. The crawlers might be written using AI but they're evidently not actually running AI inference over the pages they get back--besides being able to tell this from the behavior, if this is pretraining input, that's enormous scale, so it'd mean a large increase in effective t…

There's nothing "denialist" about recognizing the utter stupidity of systems that are being mislabeled as "AI". You judge a tool by its results, and the results have been very poor indeed. The only heads in the sand are those whose owners continually refuse to recognize the proofs before their very eyes that there's zero intelligence here.

Re: Creepy Crawlies

#320

Post by riknos314 »

calvinmorrison wrote: post #249

I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years. Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.

The amount of memory required to cache all possible diffs (defined as an ordered pair of commits) would likely be in the exobytes. At current ram prices that's easily a trillion dollars of ram to run that cache lol. Git focuses on making diff calculations efficient largely because the space of possible diffs is very expensive to enumerate.

The following from Claude: """ A diff between two randomly chosen commits usually spans years of history, so it's not a few KB — the tree itself is ~1.5 GB of text, and a multi-year span rewrites a large slice of it. Call it 100–200 MB per pair on average: 8.5×10¹¹ pairs × ~2×10⁸ bytes ≈ 10²⁰ bytes, or ~150 exabytes """

Post reply on HN