Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

391–400 of 691 posts

Re: Creepy Crawlies

#391

Post by rvz »

harshreality wrote: post #163
tptacek wrote: post #153

Earlier quoted context omitted.

I don't know what you mean by "the best argument". My entire argument is that it's bad CS. I'm not trying to convince people not to run Anubis. If anything, I'm trying to convince Anubis to use better computer science. That would make more people run it! But even that's just a faint subsidiary goal; the real goal, as someone who works in this field and has designed proof-of-work antiabuse systems before, is just to a…

I too would love anubis to use better CS, if available. Perhaps you have some ideas for @xena, instead of just letting anubis continue on the trajectory of iterating its PoW approach?

Same here. But of course and as expected; 5 hours later, the GP comment offers no proper solutions for this "better computer science" they speak of.

It shows they really don't care and they are just rage-baiting readers in their weekly grumpiness without any suggestions.

Unless they come up with said "solution", they can completely ignored in their complaints.

Re: Creepy Crawlies

#392

Post by andrewaylett »

Just for fun, because I could, I vibed up a `cgit` replacement that runs entirely in the browser -- point it at a git repo where you've run `git update-server-info` and it'll load files as if it's starting to clone the repo, using range requests and browser caching to avoid actually loading more data than necessary for the view you've requested.

I'm certainly not saying you should use this code, but it's a proof of concept for avoiding the CPU overhead of cgit rendering by loading the data on the client.

It cost me £8.27 of Fable use (from the free credits I've been given) and 56% of my five hour quota on a $20/month Pro plan. There's no server logic, it's 1.3MB of minified JS and CSS and (while I'm absolutely not suggesting anyone try to use it) it basically works: https://github.com/andrewaylett/rgitweb

This is a one-shot, my prompt set the expectation that I'd be able to load resources using CORS but (not entirely unreasonably) the Git hosts I've tried don't set CORS headers.

Shared more because I was pleasantly surprised at how cheap and easy this was -- and with a repo link because talking about it without sharing the link would be a bit crass.

Re: Creepy Crawlies

#393

Post by Barbing »

inigyou wrote: post #255
ynniv wrote: post #74

this is an increasingly common situation. it goes something like: - i have a free, niche resource - it becomes too popular - i make it more efficient - now it's really popular, and people are "abusing" it - let's make them proof-of-work - ... and proof-of-work harder - but now "legitimate" users can't use it - ??? the core problem is that the average person uses a mobile device where work is expensive, and the "attac…

What's actually stopping them isn't the PoW, it's the customisation effort. If one site has Anubis nothing scrapes it. If many sites have Anubis they write counter scrapers. Today if you make a slight change to the Anubis algorithm on your site, they'll burn CPU endlessly computing hashes with the original algorithm and submitting wrong ones. The author of Anubis hates this fact and will ban you if you mention it, so…

Searching around for this, found "make users click the mouse three times"[1] as an anti-bot idea. Generally, makes sense that smaller site owners can make small customizations to existing anti-bot tech and see positive results until they're either (1) deemed valuable enough to receive custom attention or (2) the scrapers include LLM-based anti-antibot methods.

>will ban you if you mention it

Even if mentioned really politely?

[1] would hope anyone trying this makes it accessible to visitors with disabilities

Re: Creepy Crawlies

#394

Post by jcul »

stjo wrote: post #386
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a serie…

Monero is fascinating. I downloaded the spec and tried to digest it, but the mathematics and cryptography are quite dense, at least for a casual read.

I might try to tackle it again when I can give it some more attention.

Re: Creepy Crawlies

#395

Post by p_j_w »

NavinF wrote: post #327
inigyou wrote: post #240

Earlier quoted context omitted.

It does. It's literally a felony but for some reason not a single person has pressed charges.

turns out the overlap between "people who can't configure their webserver to serve at wire speed" and "people who can get law enforcement to take them seriously" is the empty set

Didn’t read the article, huh?

Re: Creepy Crawlies

#396

Post by figglestar »

mschuster91 wrote: post #378
TeMPOraL wrote: post #267

Earlier quoted context omitted.

Where on Earth do people get the belief that: - It's the SOTA companies doing it? - Scrapers are doing it for training data ? Those are two assumptions I see in posts and threads around Anubis, that are taken at faith, and never once substantiated.

> It's the SOTA companies doing it? There are more than just the American top dogs (OAI, Anthropic, SpaceX, Facebook)... especially the Chinese government with all its infinite cash resources and next to zero ethical constraints. I don't trust the US top dogs at all, but I think the fear of discovery alone would lead them to not use "residential proxy" services. Non-US/EU entities however... who cares?

Why would they directly use a proxy service? I'd just launder the data scraping through some third party company that I could slough off if it ever turned into a news story. Not that anything would happen to them if they directly used these services anyway.

Re: Creepy Crawlies

#397

Post by Barbing »

kalkin wrote: post #355
bigstrat2003 wrote: post #319

Earlier quoted context omitted.

There's nothing "denialist" about recognizing the utter stupidity of systems that are being mislabeled as "AI". You judge a tool by its results, and the results have been very poor indeed. The only heads in the sand are those whose owners continually refuse to recognize the proofs before their very eyes that there's zero intelligence here.

> zero intelligence here That's head-in-the-sand stuff. AI is certainly very capable of being dumb (as are humans). But: > “The problem was in need of a new real idea, which this new result seems to provide,” says James Maynard, a mathematician at the University of Oxford. “It seems that the AI has made a genuinely interesting mathematical contribution.” https://www.scientificamerican.com/article/no-ai-didnt-just-...…

>AI is certainly very capable of being dumb (as are humans).

I wonder how exactly the average scraper got to be so inefficient on kernel.org.

Did someone prompt a SotA model to write the most generic scraper possible?

Did someone prompt an old local model on their laptop to write a kernel.org scraper?

Perhaps no LLMs were involved in the first place. Seems to me there isn't much relation between how good a random scraper is and how usable/effective Mythos/Sol's outputs can be.

Re: Creepy Crawlies

#398

Post by hahnbee »

robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

what constitutes as a bad crawler and what information do you have that's attracting so many bots?

Re: Creepy Crawlies

#399

Post by CqtGLRGcukpy »

I just had a look through my logs, and I've had over 90,000 requests from known AI bots over the last month. All this to a personal website that doesn't post very often.

And that's just known AI, I can't imagine what requests are pretending to a real person when they aren't.

Re: Creepy Crawlies

#400

Post by charles_f »

hahnbee wrote: post #398
robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

what constitutes as a bad crawler and what information do you have that's attracting so many bots?

Not op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers
Post reply on HN