Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

451–460 of 701 posts

Re: Creepy Crawlies

#451

Post by afdbcreid »

Aurornis wrote: post #187
wongarsu wrote: post #100

Earlier quoted context omitted.

Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Fo…

> Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time. So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views. A user who needs to load a single page and…

So, they can interleave different tokens and get stopless access at normal price?

Yes this needs to be tailored for Anubis, but that was the point: the moment it will become popular enough, scrapers will have no problems to bypass it.

Re: Creepy Crawlies

#452

Post by zbentley »

gruntled-worker wrote: post #421

There's something missing from the picture. The bots are: - Using a terribly inefficient way to redownload the same commits as e.g. HTML diffs, possibly the most inefficient. - Putting in tons of CPU cycles to surpass the Anubis PoC. - Putting in other kinds of active effort like reworking access methods and buying "residential proxies" that are probably illegal in most jurisdictions. This sounds more like escalating…

Possible, but I doubt it. The sheer number of other free-to-read content sites dealing with the exact same problem the last few years (many of whom there's no plausible reason to DDoS) tells us that this is content harvesting, not an attack.

Re: Creepy Crawlies

#453

Post by zbentley »

inigyou wrote: post #273
feelamee wrote: post #98

Earlier quoted context omitted.

The law will not work without a technical solution. So, you need both. But, at least, technical.

You just take a sample of the crawling to whichever police department knows how to deal with DDoS, they subpoena the ISPs involved and trace who's using the proxies?

> just

> [four phases, each of which has historically been hard-to-impossible for highly organized international coalitions]

Never going to happen.

Re: Creepy Crawlies

#454

Post by CGamesPlay »

zbentley wrote: post #443
CGamesPlay wrote: post #442

Git forges seem especially prone to this: tons of information, highly valuable to scrapers, rendered through several different lenses, gives a combinatoric explosion of URLs. Obviously scrapers could just be less stupid and clone the repo, but it's not happening. It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens…

That'd fail for three reasons. First, scrapers would start running JS. Whether they're running chromium-in-a-box or something more clever doesn't matter. Compute is effectively free for them, and even shitty WebOS set-top boxes can probably run a stripped-down headless browser with a JavaScript engine. Second, a full clone in browser memory is massive for something like the Linux kernel. Lots of browsers and pro-JS f…

1. Preventing access to open-source code by scrapers is not the goal behind this idea. The goal is to reduce the server load taken by the scrapers.

2. Why would you need a full clone to access /blob/cee9395acd8043be0644b25c34bfa86623f2b935/block/badblocks.c?

Re: Creepy Crawlies

#455

Post by margalabargala »

chr15m wrote: post #426
stjo wrote: post #386

Earlier quoted context omitted.

> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a serie…

Monero is fine with making people wait, that's literally the whole basis of mining blocks in "crypto currency". It's tuned to the time required. Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.

One possible way to do this: rather than simply burning hashes for nothing, Anubis can burn hashes to mine crypto. This way, the bots pay for the infrastructure they consume, at least in part.

Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.

Re: Creepy Crawlies

#456

Post by zbentley »

ButlerianJihad wrote: post #345
mbirth wrote: post #338

Earlier quoted context omitted.

Many thanks! I've just submitted the first batch of 3000 (daily limit) IP addresses.

That is a ridiculous way to try and deal with the problem of residential proxies. You are, in reality, only hurting the actual owners, the subscribers of those ISPs who are behind those addresses. We call that "collateral damage". If any of those actual residential users try to use a website, their ability to freely access the Internet may be harmed by a bad reputation that they do not deserve. They may be totally un…

> You are, in reality, only hurting the actual owners

How many times do I have to hurt them before they decide to buy a different smart TV?

Seriously, that's like saying "if you try to stop your neighborhood rodent problem by getting citations sent to people with cat food on their porch, you're just hurting the innocent outdoor cat owners". They're participating, whether they know it or not. We can and should PSA and shame and regulate away residential proxies on the supplier side, but we can and should also simultaneously discourage them on the end-user side as well.

Re: Creepy Crawlies

#457

Post by Wowfunhappy »

micah_chatt wrote: post #437
Wowfunhappy wrote: post #433

Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that? And none of them are smart enough to realize they could just `git clone` all the content and use it offline?

If you do the math (also a common system design interview question for an AI lab), its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes. At that price point, its actually very affordable to many thousands of organizations to get their own copy. I would expect the major labs to special case kernel.org…

> its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes.

That is super interesting, thank you!

> At that price point, its actually very affordable to many thousands of organizations to get their own copy.

I'm still confused as to who is actually doing it though! Maybe it's affordable to scrape and store, but training a competitive AI model is going to cost much more, right?

Re: Creepy Crawlies

#458

Post by fc417fc802 »

wongarsu wrote: post #100
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Fo…

> If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution.

Exactly, and that's even before accounting for more sophisticated PoW algorithms such as the one used by XMR that almost entirely eliminate any benefit from offloading to an accelerator.

Re: Creepy Crawlies

#459

Post by wolttam »

I think the solution is for the POW being done by the clients to *actually benefit the site owner*. Users remain just as mildly annoyed as with Anubis, but maybe a bit less knowing that the work they’re doing benefits the site owner/author, and the system helps thwart the bots (or at least makes them do work that benefits the author).

Re: Creepy Crawlies

#460

Post by zbentley »

boredatoms wrote: post #369

Can I suggest putting some text in the page that tells the bot what the more efficient download method is?

What makes you think that would accomplish anything?

These scrapers aren't LLM agents. They're distributed classical programs that harvest data which is later used to train an LLM. The LLM doesn't write the scraper or respond to individual scrape events. The entity training the LLM contracts someone, who contracts someone, who contracts someone to run a web scraper and send them the data.

Post reply on HN