Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

331–340 of 660 posts

Re: Creepy Crawlies

#331

Post by jopsen »

dspillett wrote: post #226
jopsen wrote: post #177

Earlier quoted context omitted.

Probably they must be deduplicating text they've seen before. The only punishment would be unique text that trains their models to be degenerate. And even then you'd probably have to serve across many domains.

One thing I've thought of trying is, if I ever get around to putting things back online, loading the pages with altered versions of themselves in a manner that human users are unlikely to see (though they could if they wanted to) but scrapers are unlikely to ignore (because they look enough like something users might see, perhaps by opening a tag). The altered versions would be the page or paragraphs of it, with sent…

To be clear, making pages with bad training data for bots won't make the bots go away.

It'll just punish the bad actors running the scrapers. As the original poster mentions that they are using TVs as proxies to get residential IPs, one really can't think of these bots as criminal enterprises.

Sadly, if the bad actors has two cents for brain, they'll limit how much importance each domain name can have on training data. To mitigate impact of bad data like this.

(Note I'd suggest only linking to them from robot.txt as pages to not be indexed, that way no human or well behaved not ever will see them, which is kind of the point).

Re: Creepy Crawlies

#333

Post by tptacek »

progval wrote: post #166
tptacek wrote: post #118

Earlier quoted context omitted.

This is the basic misconception behind the Anubis design. In an anti-spam system, there's a user/abuser reward asymmetry. Users make single requests and get some constant reward V. Abusers make millions of requests and also get some constant reward V. Proof-of-work exploits the asymmetry: costs are, by a factor of ~millions, disproportionately imposed on abusers. That's exactly not what happens with scrapers. Scraper…

> There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers. There is: users solve the PoW once, scrappers solve it every time they rotate IP addresses. Some of them change their IP address on every request, so they keep solving the PoW over and over.

That doesn't matter! Every time they successfully solve the challenge, they're getting the token reward. That's the missing asymmetry. In an anti-spam system, 99.999% of all transactions are unrewarded.

Re: Creepy Crawlies

#334

Post by someothherguyy »

robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

wouldn't that just make your connection load worse?

Re: Creepy Crawlies

#335

Post by tptacek »

xena wrote: post #325
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

The WebAssembly challenge (as of yet unreleased, it's been in dev for a year and I don't want to enable it by default until I know it won't blow up horribly) is bit-wise instead of nibble-wise, meaning that admins have better challenge granularity: https://anubis.techaro.lol/docs/admin/configuration/challeng...

What I'm hearing is that you spent a year designing a more complicated proof-of-work challenge rather than moving to a challenge system that actually impacts scrapers. Consider revisiting this decision!

Re: Creepy Crawlies

#336

Post by jay_kyburz »

inigyou wrote: post #280
NooneAtAll3 wrote: post #60

Earlier quoted context omitted.

it's hard to separate spambot that only accesses 3-5 links per IP and a legit user. Changing content for legit user can be devastating

Put a cookie wall in front. The bot will either load the cookie and have a persistent identifier, or not load the cookie and not get in

why is this not the answer? then you can also rate limit each cookie as well.

Re: Creepy Crawlies

#337

Post by embedding-shape »

wongarsu wrote: post #100
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Fo…

> which takes ~180sec for my iPhone 17 to solve at ~100KH/s

> so I really don't see why Safari on a modern iPhone should be so slow at this

FWIW, my iPhone 12 Mini also does ~110KH/s with Anubis on lists.ffmpeg.org, so seems fairly likely that Safari somehow here isn't working as expected.

Re: Creepy Crawlies

#338

Post by mbirth »

inigyou wrote: post #268
mbirth wrote: post #194

Earlier quoted context omitted.

I’d love a service like spamcop.net where I could submit my access_log and they lookup the abuse addresses and file abuse reports in my name. Maybe if people’s Internet access gets suspended they’ll think about installing random apps that work as a proxy in the background.

abuseipdb.com Some ISPs ban customers based on a single report there - have fun!

Many thanks! I've just submitted the first batch of 3000 (daily limit) IP addresses.

Re: Creepy Crawlies

#339

Post by roughly »

inigyou wrote: post #248
marginalia_nu wrote: post #114

Earlier quoted context omitted.

It's very likely the last few years of bot behavior is the consequence of the residential proxy business booming. This is indirectly due to AI company crawling, but the fact that they are as cheap and available as they are changes the incentives for anyone using them toward reckless and unsustainable request behavior, as there is no risk of burning your IPs, and very small chances of seeing any consequences of essent…

And the residential proxy business was created by Cloudflare, who was created by us using Cloudflare. I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I…

Wait, so you’ve been the person who wanted to keep people from scraping your site, the person who’s trying to scrape your site, and the person getting paid to help someone scrape your site? Brother, what are you doing with your life?

Re: Creepy Crawlies

#340

Post by dspillett »

jopsen wrote: post #331
dspillett wrote: post #226

Earlier quoted context omitted.

One thing I've thought of trying is, if I ever get around to putting things back online, loading the pages with altered versions of themselves in a manner that human users are unlikely to see (though they could if they wanted to) but scrapers are unlikely to ignore (because they look enough like something users might see, perhaps by opening a tag). The altered versions would be the page or paragraphs of it, with sent…

To be clear, making pages with bad training data for bots won't make the bots go away. It'll just punish the bad actors running the scrapers. As the original poster mentions that they are using TVs as proxies to get residential IPs, one really can't think of these bots as criminal enterprises. Sadly, if the bad actors has two cents for brain, they'll limit how much importance each domain name can have on training dat…

> … making pages with bad training data for bots won't make the bots go away. It'll just punish the bad actors running the scrapers.

Exactly. I can't hope to keep them all at bay, but I can at least have the petty little victory of making their visit less convenient than it might otherwise be.

> if the bad actors has two cents for brain

I suspect that a majority of them are little better than the script kiddies of yore, running tools with minimal understanding of what is actually going on.

> I'd suggest only linking to them from robot.txt as pages to not be indexed

Agreed. Blocking all bots from all pages, well those that bother to listen to robots.txt. All bots because pretty much all of them are scraping for AI and similar these days, even googlebot. If I want people to see my stuff they'll get a link, and maybe they'll pass it on further, but all indexers/trainers can get stuffed. I'll likely make an exception for archive.org and similar.

Post reply on HN