61–70 of 968 posts
>This dance to get access is just a minor annoyance for me, but I question how it proves I’m not a bot. These steps can be trivially and cheaply automated. >I think the end result is just an internet resource I need is a little harder to access, and we have to waste a small amount of energy. No need to mimic the actual challenge process. Just change your user agent to not have "Mozilla" in it; Anubis only serves you…
[dead]
The security policy that didn't exist until a few hours ago?
Copied to the root of the repo after the disclosure
ref: https://github.com/TecharoHQ/anubis/issues/1002#issuecomment...
>This dance to get access is just a minor annoyance for me, but I question how it proves I’m not a bot. These steps can be trivially and cheaply automated. >I think the end result is just an internet resource I need is a little harder to access, and we have to waste a small amount of energy. No need to mimic the actual challenge process. Just change your user agent to not have "Mozilla" in it; Anubis only serves you…
[flagged]
But the project description describes it as a project to stop AI crawlers?
> Weighs the soul of incoming HTTP requests to stop AI crawlers
What exactly is so bad about AI crawlers compared to Google or Bing? Is there more volume or is it just "I don't like AI"?
either way the result is the same: they induce massive load
well written crawlers will:
- not hit a specific ip/host more frequently than say 1 req/5s
- put newly discovered URLs at the end of a distributed queue (NOT do DFS per domain)
- limit crawling depth based on crawled page quality and/or response time
- respect robots.txt
- make it easy to block themEarlier quoted context omitted.
The security policy that didn't exist until a few hours ago?
Added on March 18: https://github.com/TecharoHQ/.github/commits/main/SECURITY.m... Copied to the root of the repo after the disclosure ref: https://github.com/TecharoHQ/anubis/issues/1002#issuecomment...
Earlier quoted context omitted.
> AI companies use residential proxies Source:
Source: Cloudflare https://blog.cloudflare.com/perplexity-is-using-stealth-unde... Perplexity's defense is that they're not doing it for training/KB building crawls but for answering dynamic queries calls and this is apparently better.
However, if this information is accurate... perhaps site owners should allow AI/bot user agents but respond with different content (or maybe a 404?) instead, to try to prevent it from making multiple requests with different UAs.
Earlier quoted context omitted.
>No. A human sees a 10x slowdown. For the actual request, yes. For the complete experience of using the website not so much, since a human will take at least several seconds to process the information returned. >And the scraper paid one 1/1000000th of a dollar. (The scraper does not care about latency.) The point need not be to punish the client, but to throttle it. The scraper may not care about taking longer, but t…
A proof of work challenge does not throttle the scrapers at steady state. All it does is add latency and cost to the first request.
Earlier quoted context omitted.
The security policy that didn't exist until a few hours ago?
Added on March 18: https://github.com/TecharoHQ/.github/commits/main/SECURITY.m... Copied to the root of the repo after the disclosure ref: https://github.com/TecharoHQ/anubis/issues/1002#issuecomment...
I really don't understand the hostility towards the mascot. I can't think of a bigger red flag.
Out of curiosity, what did you read as hostility?
What exactly is so bad about AI crawlers compared to Google or Bing? Is there more volume or is it just "I don't like AI"?
As others have said, it's definitely volume, but also the lack of respecting robots.txt. Most AI crawlers that I've seen bombarding our sites just relentlessly scrape anything and everything, without even checking to see if anything has changed since the last time they crawled the site.
Why this is the case while web-crawlers have been scrapping the web for the last 30 years is a mystery to me. This should be a solved problem. But it looks like this field is full of wrongly behaving companies with complete disregards toward common goods.