One problem with Anubis is that once you've solved the POW once, you just need to hold the cookie to avoid solving it again. Scrapers have probably learnt to do that by now. So Anubis isn't as effective as it used to be before it was widely used.
Creepy Crawlies
431–440 of 702 posts
Re: Creepy Crawlies
#432$1 dollar a year subscriptions would help.
Re: Creepy Crawlies
#433Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that?
And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
Re: Creepy Crawlies
#434Earlier quoted context omitted.
Real attackers fuck. If they have IP address control issues they will fix them. The bots being stopped are the silly ones.
Okay. That's good enough. The problem people using Anubis have is not "I'm being targeted for a DDoS" and is actually "my site is being incidentally DDoS'd because scrapers are DDoSing the entire internet". A solution to the latter is what people need . Every single one of these threads comes up with hypotheticals about how useless Anubis is and how it can't possibly work and yet it does work for the real-world probl…
If every site does it the scrapers will adapt. The scrapers will always have a cost advantage over real users for this.
Re: Creepy Crawlies
#435Earlier quoted context omitted.
Same here. But of course and as expected; 5 hours later, the GP comment offers no proper solutions for this "better computer science" they speak of. It shows they really don't care and they are just rage-baiting readers in their weekly grumpiness without any suggestions. Unless they come up with said "solution", they can completely ignored in their complaints.
I had a fairly long conversation with Xe privately about this and didn't feel the need to repeat it here, especially since (as the search box will tell you) I've already talked about this before. But, do go on.
I looked at your comments and there's a bunch saying that Anubis is bad CS but none I could find saying what, exactly, you want Anubis to do instead. You sure are posting a lot about how it's bad, though.
Re: Creepy Crawlies
#436Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that? And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
There are many more labs than that, and humans aren't designing unique scraping processes per domain.
Re: Creepy Crawlies
#437Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that? And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
If you do the math (also a common system design interview question for an AI lab), its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes. At that price point, its actually very affordable to many thousands of organizations to get their own copy. I would expect the major labs to special case kernel.org similarly to other sites like Wikipedia, but not the majority of scrapers
Re: Creepy Crawlies
#438Re: Creepy Crawlies
#439There has to be some not-yet-discovered way to have a capcha that is easy for any human but impossible for robot. Too bad capchas hurt user experience no matter how easy they are. Another solution I came up with while reading HN comments: whitelist IPs instead blacklisting. Give access to well-behaving hosts/groups. It can even be shared across different sites. Although this would create a market for selling "good IP…
> whitelist IPs instead blacklisting
I think I agree with this. There will be difficulties though:
- Will likely have to be pay-to-play. No point in whitelisting IPs if you are bombarded with whitelist requests by a bot.
- Will come with the assurance that any bot traffic will result in an IP ban.
- Will rule out people with dynamic IP addresses.
- Will be difficult to gain new users.
Re: Creepy Crawlies
#440In the case of kernel.org, why not make the unauthenticated version return only the latest kernel repo with no history (tiny number of URLs relatively speaking). If you want full kernel.org features, login.
because they do not want to be twitter, reddit, facebook, ...
I'm not suggesting 'login' because facebook/twitter/etc, just as a mechanism to make the bot problem go away. They clearly want it to stop, they tried obvious methods but the AI platforms are circumventing it (deliberately) which is poor form.