Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new AI traffic options for customers

blog.cloudflare.com

Post reply on HN
Print view View original

91–100 of 169 posts

Re: Cloudflare's new AI traffic options for customers

#91

Post by inigyou »

nirui wrote: post #54
usef- wrote: post #32

Earlier quoted context omitted.

How are they playing both sides? I thought their scraping products were also about having it behave and not take down systems

> not take down systems "Block on pages with ads" is probably about preventing the AI crawlers from clicking on the ads which maybe considered cheating by the ad company. If you want to prevent "bot attacks", maybe the "Block" option will do the trick. But of course, to do all that you need to put some trust on Cloudflare, because they're the one identifying the bots from normal users. For me, as someone who's hostin…

> If you want to prevent "bot attacks", maybe the "Block" option will do the trick.

This morning, for the first time, I clicked on a link posted on HN and was told in no uncertain terms that I am a bot and will not be allowed to view this page. By Cloudflare.

Your strategy is fine and similar to one of the checks in go-away. The point is that the unknown global DDoS adversary is using a very simple scraper, which does not load images or CSS or scripts, does not set cookies, etc

Re: Cloudflare's new AI traffic options for customers

#92

Post by weird-eye-issue »

remus wrote: post #67
weird-eye-issue wrote: post #53

Earlier quoted context omitted.

If a site cannot handle traffic from the real Googlebot that is a serious issue with the site itself since it's actually pretty conservative Also I should note there are lots of fake Googlebots...

Indeed, I've got a site which gets a lot of bot traffic and google bot is pretty sensible compared to a lot of other mainstream bots.

Yes, it really does not make that many requests. In fact lots of site owners struggle with having it not crawl and index their site enough

Re: Cloudflare's new AI traffic options for customers

#93

Post by inigyou »

prologic wrote: post #46
fc417fc802 wrote: post #19

Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go…

PoW schemes like Anubis don't work. Increasingly bots are using headless browsers and are basically able to solve captchas, proof-of-work(s) and basically bypass all any any attempts to block them. It's becoming impossible to stop bots from hammering your sites/services for unwanted traffic.

There's one or a small number of actors who are DDoSing the internet right now, and those are the ones you care about. What's the purpose for blocking non-DDoS bots?

Re: Cloudflare's new AI traffic options for customers

#94

Post by inigyou »

gruez wrote: post #25
fc417fc802 wrote: post #19

Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go…

>I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. ??? Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. b…

Yesterday Cloudflare told me I was a bot and couldn't access this.

Re: Cloudflare's new AI traffic options for customers

#95

Post by inigyou »

gruez wrote: post #35
gucci-on-fleek wrote: post #29

Earlier quoted context omitted.

> And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge. > If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's. Cloudflare blo…

>You are guaranteed to pass an Anubis challenge eventually [0] That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap. >Cloudflare blocks legitimate users itself sometimes [1]. I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.

The bots that are problems right now are not the ones that run Anubis challenges. There is no universal anti-bot, only ones that work at certain times.

Re: Cloudflare's new AI traffic options for customers

#96

Post by inigyou »

gucci-on-fleek wrote: post #45
gruez wrote: post #35

Earlier quoted context omitted.

>You are guaranteed to pass an Anubis challenge eventually [0] That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap. >Cloudflare blocks legitimate users itself sometimes [1]. I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.

> That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap. Yeah, I really have no idea why Anubis works right now: residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs, but they give up on even short-ish Anubis challenges. > I never ran into this issue despite using seemingly ma…

Because the (singular) global ddos adversary is using a very dumb scraper, even dumber than wget --mirror

Re: Cloudflare's new AI traffic options for customers

#97

Post by inigyou »

bornfreddy wrote: post #84
m00dy wrote: post #59

Earlier quoted context omitted.

Not sure why Anubis is getting so much hype on HN, but honestly, it is not the solution. A real solution would use behavioral modeling. Most browser fingerprinting issues are already largely solved anyway.

As someone who is regularly blocked by CF and G - no, they are not. Even worse, they should not be, because that would mean the complete end of privacy online (not that we are far from that). Anubis works because at scale it wastes bots' resources (time mainly).

it works because the global ddos adversary doesn't run javascript

Re: Cloudflare's new AI traffic options for customers

#98

Post by inigyou »

jerf wrote: post #42
holografix wrote: post #26

What’s the end goal for Cloudflare and the web here? I don’t think ADOG (anthropic, deepmind, openai, google) is going to pay to crawl. What would force their hand? It’s more likely they’ll strike undisclosed agreements with major sources of discussion like reddit etc. That’s not to say getting new information as a way of context-providing is not going to happen but that’s not scraping.

I think the best answer is, nobody knows. The previous equilibrium for content scraping for search engines on the internet was already at times an uncomfortable one. But I agree that from a game theory perspective, "the AI bots take and give nothing back in return" is not just hyperbole, it's the actual situation. If Google is successful in what seems to be its plans and it becomes a box where you type a question and…

The internet becomes full of free propaganda since it's not the consumer who pays for that?

Re: Cloudflare's new AI traffic options for customers

#99

Post by inigyou »

Fizz43 wrote: post #52

>So, instead of defining a bot primarily as “AI” or not, our updated approach to classification will ask deeper questions about bot or agent behavior: What are they doing on my site? What are they storing? And how will they reshare my content? I dont get this. The question is are they a bot or a human. It doesnt matter what they are doing I dont want bots on my site.

Why not. You need a reason. A browser is just a bot that renders pages.

Re: Cloudflare's new AI traffic options for customers

#100

Post by inigyou »

PeterStuer wrote: post #78
arjie wrote: post #18

This is fine so long as it’s easy for me to turn off. I just don’t want to accidentally lose all AI traffic one day.

Most of the internet unfortunatly ploinks fown a 'free' service and never even looks at what it defaults to. E.g., quite a lott of cloudflare "protected" sites block their rss feeds from being read by machine.

This is deliberate from Cloudflare. It wants to block Google's access to as much of the internet as possible.
Post reply on HN