Earlier quoted context omitted.
What's the "honors system"?
Robots.txt
Bingo!
Cloudflare's new AI traffic options for customers
31–40 of 169 posts
Re: Cloudflare's new AI traffic options for customers
#32> For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default. It's kind of exhausting seeing Cloudflare playing both sides of the arms race. I just can't imagine bringing myself to use their technology to build agents and build AI products when they're also doing things like this. > This also…
How are they playing both sides? I thought their scraping products were also about having it behave and not take down systems
Re: Cloudflare's new AI traffic options for customers
#33Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go…
>I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. ??? Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. b…
Try using Firefox with "resist fingerprinting" enabled, on OpenBSD, if you want an idea of how insidious Cloudflare's increasing prevalence is for the openness of the web. Inexplicable 403 responses for entire domains for no apparent reason, no "captcha", just blocked outright. Heck, it happens on Linux too, and even without "resist fingerprinting" enabled.
Re: Cloudflare's new AI traffic options for customers
#34What’s the end goal for Cloudflare and the web here? I don’t think ADOG (anthropic, deepmind, openai, google) is going to pay to crawl. What would force their hand? It’s more likely they’ll strike undisclosed agreements with major sources of discussion like reddit etc. That’s not to say getting new information as a way of context-providing is not going to happen but that’s not scraping.
I get a mixed feeling about all this. Cloudflare is unilaterally making all these decisions which impact the whole internet traffic flow. Taking the lead is one thing, however decisions like this should have the direct involvement of Internet Engineering Task Force (IETF) to account for all stakeholders, otherwise we run into the situation of a fragmented internet
I'm curious if this is just to pressure Google into separating their crawlers
Re: Cloudflare's new AI traffic options for customers
#35Earlier quoted context omitted.
>I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. ??? Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. b…
> And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge. > If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's. Cloudflare blo…
>You are guaranteed to pass an Anubis challenge eventually [0]
That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap.
>Cloudflare blocks legitimate users itself sometimes [1].
I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
Re: Cloudflare's new AI traffic options for customers
#36Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go…
> Please consider installing one of the many PoW schemes such as anubis
Why not go all the way and mine monero instead of just completely wasting the work?
Re: Cloudflare's new AI traffic options for customers
#37Re: Cloudflare's new AI traffic options for customers
#38So, in summary: still the honors system. Got it. thanks.
What's the "honors system"?
An honor system is a system without any (explicit) external enforcement of rules. For example an unnattended fruit stand, where people are trusted to be honorable and leave payment for the fruit they take.
Re: Cloudflare's new AI traffic options for customers
#39Earlier quoted context omitted.
I get a mixed feeling about all this. Cloudflare is unilaterally making all these decisions which impact the whole internet traffic flow. Taking the lead is one thing, however decisions like this should have the direct involvement of Internet Engineering Task Force (IETF) to account for all stakeholders, otherwise we run into the situation of a fragmented internet
I'm curious if this is just to pressure Google into separating their crawlers
This is more of a way to create a unauthorized toll tax on highway. There is a problem indeed, however if the proposed direction by cloud flare is to solve it or benefit out of it is a debatable topic.
Re: Cloudflare's new AI traffic options for customers
#40Earlier quoted context omitted.
>I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. ??? Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. b…
Try using Firefox with "resist fingerprinting" enabled, on OpenBSD, if you want an idea of how insidious Cloudflare's increasing prevalence is for the openness of the web. Inexplicable 403 responses for entire domains for no apparent reason, no "captcha", just blocked outright. Heck, it happens on Linux too, and even without "resist fingerprinting" enabled.
Happens with older chrome browsers as well. Last supported windows 7 version. Started this month.