81–90 of 169 posts
So is it possible to say "No bots except Google, OpenAI, Grok, Claude and Perplexity"? As far as I can tell, Google is the only one sending me visitors. And the other big AI players might do so in the future. Another option would be "No anonymous bots". So at least if a bot would want to crawl my site, they would have to identify themselves. Since the rise of the AI bots, I am getting hurt badly with insane amounts o…
Earlier quoted context omitted.
How are they playing both sides? I thought their scraping products were also about having it behave and not take down systems
I mean that they're telling developers that they should use Cloudflare's platform to build agents, the kind of agents that would go across the web and act on behalf of users... but then they're also the ones blocking those requests. This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled wi…
Earlier quoted context omitted.
Good. Google's approach here is manifestly predator, unfair, and IMO illegal. They deserve to be in court for this behaviour, and mandating owners give consent for AI training or drop out of Google; which is just a non-starter because they're a search monopoly. That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a tr…
I don't think it's the Google bot DDOSing people's infrastructure for AI training...
Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go…
Not sure why Anubis is getting so much hype on HN, but honestly, it is not the solution. A real solution would use behavioral modeling. Most browser fingerprinting issues are already largely solved anyway.
>So, instead of defining a bot primarily as “AI” or not, our updated approach to classification will ask deeper questions about bot or agent behavior: What are they doing on my site? What are they storing? And how will they reshare my content? I dont get this. The question is are they a bot or a human. It doesnt matter what they are doing I dont want bots on my site.
> I dont get this. The question is are they a bot or a human. It doesnt matter what they are doing I dont want bots on my site. Do you want your site to be discoverable by a search engine? (How do you think that occurs?)
That being said, the issue with it right now is that Google uses the same crawler for both AI training and search, so this step puts a small pressure on google nonetheless to hopefully split them.
Earlier quoted context omitted.
A CDN doesn't necessarily have to perform a MitM. We really need more nuanced terminology to distinguish the various approaches.
Right, but practically speaking all CDNs are MITMs. If you're against cloudflare you should be against cloudfront, akamai, etc. as well.
Akamai, Fastly, etc only take big customers who know what they're doing. You need to sign a proper contract with them. They aren't low-friction.
> For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default. It's kind of exhausting seeing Cloudflare playing both sides of the arms race. I just can't imagine bringing myself to use their technology to build agents and build AI products when they're also doing things like this. > This also…
How are they playing both sides? I thought their scraping products were also about having it behave and not take down systems