31–40 of 178 posts
Earlier quoted context omitted.
Sure, then just replace it with something that is useful to society but not immediately profitable to a scraper, like science research
I want to do this eventually, but it's hard to split things into the micro-tasks that would be required to make this work on Anubis. One of the ideas I'm throwing around is a world where Anubis helps fuzz old games to find timesaves in tool-assisted speedruns. It's harder than you think.
Love this. There’s been past discussion on HN re how OSS maintainers are treated, and this is such a wry sentence. I really appreciate the tone / attitude to the problem.
It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute. Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amou…
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.
Earlier quoted context omitted.
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute. Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amou…
It’s not seconds though, it’s in the order of tens of milliseconds of work, it’s faster to complete the challenge than most http requests will take. last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16) This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.
Earlier quoted context omitted.
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute. Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amou…
It’s not seconds though, it’s in the order of tens of milliseconds of work, it’s faster to complete the challenge than most http requests will take. last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16) This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.