Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new AI traffic options for customers

blog.cloudflare.com

Post reply on HN
Print view View original

111–120 of 169 posts

Re: Cloudflare's new AI traffic options for customers

#111

Post by inigyou »

siva7 wrote: post #104
inigyou wrote: post #90

Earlier quoted context omitted.

Their scraping products are mostly about paying them money to not be blocked.

So Google has to pay Cloudflare 10B$ to get Googlebot moved to their default Allowlist otherwise Google would loose access to a significant portion of the internet? Genius move.

Yes literally this. Every large corporation gatekeeper does this sort of thing, it's why we have to militantly push for decentralisation by avoiding these corporations. It's not morally different from Google forcing your site to be AI-scrapeable or be kicked off Google Search. It's why Apple won't let you publish an app without paying them 30% of your total revenue. It's why Microsoft threatened to ban Steam (they later decided not to) so they could get all those games onto the Microsoft Store, and in response Valve ported gaming to Linux.

Re: Cloudflare's new AI traffic options for customers

#112

Post by inigyou »

dannyw wrote: post #43
simonw wrote: post #4

The big news here is that Googlebot will be blocked from September 15th onwards by one the "block training" policies, because Google use the same crawler infrastructure for their search index AND for training Gemini: > Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line…

Good. Google's approach here is manifestly predator, unfair, and IMO illegal. They deserve to be in court for this behaviour, and mandating owners give consent for AI training or drop out of Google; which is just a non-starter because they're a search monopoly. That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a tr…

How come you say "Good." and you are near the top of the comments but I say "Good." and get flagdead?

Re: Cloudflare's new AI traffic options for customers

#113

Post by inigyou »

PunchyHamster wrote: post #74
stingraycharles wrote: post #69

Earlier quoted context omitted.

I thought the issue people have with AI scrapers is the ones that DDoS sites to scrape everything for training purposes, rather than the ones that interactively query specific content to support an active conversation?

First one is problem because it brings no traffic back and occasionally DDoSes the site Second one is problem because it DDoSes the site. "Just" queries for hundred thousand people (if you happened to be good source for that bit of knowledge) that don't bring actual people to your site is also a problem

First one is not occasional, it is the anonymous global adversary probably ddosing your site right now. It never relents.

Re: Cloudflare's new AI traffic options for customers

#115

Post by inigyou »

gib444 wrote: post #70
fc417fc802 wrote: post #65

Earlier quoted context omitted.

You are asserting that they don't work without explanation or evidence. Meanwhile it isn't clear why they wouldn't and indeed they appear to accomplish the stated goal of severely rate limiting scrapers.

> You are asserting that they don't work without explanation or evidence. Oh, no, that isn't how it works — you made a claim first (albeit indirectly), with no evidence. It's on you

Evidence: they solved many people's problems

Re: Cloudflare's new AI traffic options for customers

#116

Post by inigyou »

usef- wrote: post #34
sandeepkd wrote: post #30

Earlier quoted context omitted.

I get a mixed feeling about all this. Cloudflare is unilaterally making all these decisions which impact the whole internet traffic flow. Taking the lead is one thing, however decisions like this should have the direct involvement of Internet Engineering Task Force (IETF) to account for all stakeholders, otherwise we run into the situation of a fragmented internet

I'm curious if this is just to pressure Google into separating their crawlers

That wouldn't make any sense from anyone's perspective. They would just pretend to separate them and not.

Re: Cloudflare's new AI traffic options for customers

#118

Post by dijksterhuis »

inigyou wrote: post #112
dannyw wrote: post #43

Earlier quoted context omitted.

Good. Google's approach here is manifestly predator, unfair, and IMO illegal. They deserve to be in court for this behaviour, and mandating owners give consent for AI training or drop out of Google; which is just a non-starter because they're a search monopoly. That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a tr…

How come you say "Good." and you are near the top of the comments but I say "Good." and get flagdead?

it’s likely to do with the fact that the parent comment here laid out a thoughtful basis / argument for their “good”, providing some detail to their justification for it.

that’s just my take/feedback, take it or leave it. i won’t be engaging further as i already feel i’m going against the site guidelines with this!

Re: Cloudflare's new AI traffic options for customers

#119

Post by microtonal »

fc417fc802 wrote: post #63
m00dy wrote: post #59

Earlier quoted context omitted.

Not sure why Anubis is getting so much hype on HN, but honestly, it is not the solution. A real solution would use behavioral modeling. Most browser fingerprinting issues are already largely solved anyway.

PoW is a reasonable solution as a fallback when other metrics flag a client.

How? First, they can solve the Anubis challenge with native code, so they can solve them faster than genuine users. Second, the cost is nothing compared to training LLMs, plus they will just move the work to the residential proxies that they have access to (so, someone is paying through their TV's electricity bill).

Anubis only work(s|ed) great for a while when crawlers were not prepared for these challenges. Security through obscurity.

Re: Cloudflare's new AI traffic options for customers

#120

Post by pluc »

timpera wrote: post #108

Blocking bots should not be the default behavior, it should be opt-in. Is Cloudflare trying to play both sides in order to secure a fee on every pay-to-crawl in the future?

Cloudflare's business model is playing both sides. They sell anonymity to attackers and mitigation/protection to the victims.
Post reply on HN