Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

271–280 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#271

Earlier quoted context omitted.

> Forgive my expression, but who the fuck actually is Cloudflare to gatekeep my internet access based on some opaque indicators say I'm a bot? Cloudflare is in no way gatekeeping your internet access. Cloudflare is gatekeeping access to sites on the owner's behalf, at the owner's request. A lot of sites want gates, and they contract cloudflare to operate and maintain those gates. If it wasn't cloudflare it would be s…

> Those site owners are fine with leaving the details of who to allow and deny to cloudflare, hence they contracted cloudflare to do it on their behalf And you think that giving someone this power without actual oversight is okay? It really isn't. > ranting about Schlage for gatekeeping your access to shelter.... the onwer of the building chose to have locks and picked a vendor rather than making their own Except the…

> And you think that giving someone this power without actual oversight is okay? It really isn't.

Who is overseeing who in your scenario? I think the decision is up to the company doing the contracting. They get to choose how to handle it - if they don't like the results, operations or anything else about Cloudflare they should cancel the contract and get a new vendor. If they are fine with those and want to keep it, they can do that too.

> Except they randomly find some people's "key" incorrect without giving them any recourse.

If my apartment key doesn't work, I don't contact Schlage, I contact the rental company. They may send a new key, or fix the door/lock, and even work with Schlage to fix some root problem. My contact point is still only the company I have a relationship with.

Of course the analogy breaks down here - because in the public web case it's often more like the door to a grocery store. If that is stuck locked and the store can't open, you contact the store - they'll work with their maintenance and vendors to let you in. Until its fixed they just say "sorry you don't get in", and maybe they decide to ban you for making trouble (not good business, but the store gets to do that if they want).

Lets stick with that example and generalize it to all places of business. Plenty of them have security that can ask you to leave and refuse you entry. Bars have bouncers, mall have "cops", office buildings have receptionists and "cops" - in any of those cases they can ask you to leave the premesis, or prevent you from entering the premesis and they don't have to tell you why or give you a course to remedy it. Why do you expect cloudflare to tell you why you can't access a business that doesn't want your traffic?

If you can't get to a site, contact the site owner and ask for them to figure out how to let you in - they may say no, they may tell you that they don't care if they get your traffic, or the may tell you that they'll contact cloudflare and maybe you'll see a resolution.

> Except you don't get a warning before you actually try to enter. It can be added at any point.

Again - a company can refuse your business or your entry, and they don't have to warn you in advance or tell you why. They can even change their rules without warning or explanation. If you have some sort of business with them, and they want to continue it, you have all sorts of recourse - you can call them, get a lawyer to send threatening letters or sue them, or stop paying them since they aren't fulfilling their end of the contract. Your only contract with random public websites is the HTTP protocol - even that has all sorts of "reject without explanation" options - sure they could set up error codes correctly, or just always return 500 or whatever.

> In the end, sure you might have that freedom to restrict as you wish, but someone shouldn't be doing it at this scale without informing people and without oversight.

Someone shouldn't be providing a service that people want for their sites? There can't be a business that helps people who don't want your traffic to actually reject your traffic?

Again who is overseeing who? The site owner is allowed to reject your traffic - either they don't want your traffic or they don't care if they don't get your traffic. The owners have done a cost-benefit analysis and have decided the cost of your traffic does not outweigh the benefit of using Cloudflare to reject it. I don't see how this is Cloudflare's fault.

It seems to me that you've been deemed as "not worth the hassle" and that sucks for you. I just don't see that makes Cloudflare the bad guy - if you actually are worth the hassle, talk to the people responsible for the site about why you are worth the hassle and get them to make the situation right, they are the ones who hired cloudflare and decided you weren't worth the hassle to begin with. They are the ones who can change their setting or their vendor or whatever, not the company that was hired to execute a contract on the site owner's behalf.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#272

How long does the world-wide-web have left? It's always felt like it would be around forever, but at some point it will fade into obscurity like IRC has done. The golden age, I feel, has been gone a while, but "AI" seems like the beginning of the end.

"AI" is the beginning of the end the same way as spam, malware and bot content were perceived in the past. To every action there is a reaction and "AI" won't be an exception.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#273

Earlier quoted context omitted.

> Those site owners are fine with leaving the details of who to allow and deny to cloudflare, hence they contracted cloudflare to do it on their behalf And you think that giving someone this power without actual oversight is okay? It really isn't. > ranting about Schlage for gatekeeping your access to shelter.... the onwer of the building chose to have locks and picked a vendor rather than making their own Except the…

> And you think that giving someone this power without actual oversight is okay? It really isn't. Who is overseeing who in your scenario? I think the decision is up to the company doing the contracting. They get to choose how to handle it - if they don't like the results, operations or anything else about Cloudflare they should cancel the contract and get a new vendor. If they are fine with those and want to keep it,…

> I think the decision is up to the company doing the contracting.

The sum of all websites contracting CF can be more damaging to me as an individual than it is to the companies doing the contracting. So I should definitely have a say on how they operate.

> If my apartment key doesn't work, I don't contact Schlage, I contact the rental company.

Yeah, except with your analogy it's Schlage that's breaking or fixing your keys, not the rental company. You don't know why or now. Some days it takes more effort than others to open your door. Your option is of course to move apartments, but you can't boycott Schlage, if the landlord decides to use them.

> Why do you expect cloudflare to tell you why you can't access a business that doesn't want your traffic?

Because for example if it's based on my native language or religious preferences it would be literally illegal in real life.

> Someone shouldn't be providing a service that people want for their sites? There can't be a business that helps people who don't want your traffic to actually reject your traffic?

Yeah, they should not be able to do so without someone overseeing that they aren't blocking accessibility-oriented browsers or discriminating based on non-technical factors that just happen to correlate in some other way.

> It seems to me that you've been deemed as "not worth the hassle" and that sucks for you.

I hope you get to enjoy kafkaesque technical obstacles thrown at you for no fault of your own, and I hope that sucks for you.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#274

Earlier quoted context omitted.

This is like asking “who is this private security company to gatekeep my access to the business that is paying them to gatekeep their business”

Except it's some random company picking me for "extra checks" for no specified reason, and I don't even get a warning that there's this entity there.

There's literally a guard standing at the door. You are free to leave / not visit the site. And nobody owes you an explanation for the security practices of the business that you want to patronize.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#275
post #141

Earlier quoted context omitted.

It varies a lot based on what I'm doing. Sites that rely on ads like english-language¹ recipes or health information have a lot of "you're European so you're blocked altogether" or "let me check that the connection is secure , ah wait, here is a captcha for you to solve" pages. Anything that needs to do fraud detection usually hates me as well, perhaps because I have a phone number and bank account from another count…

> That German ISPs have daily-rotating IP addresses This is interesting. What is the purpose? Security? Privacy?

Whatever the reason, it's not unilaterally true. I've had the same IP for years on a normal consumer cable internet connection.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#276
post #141

Earlier quoted context omitted.

FWIW, it can't be cookies alone that gives you an inordinate number of bot challenges. I use private tabs on Firefox (for Linux and Android) for most of my browsing, and I rarely get any challenges regardless of what I do. The only issues tend to be when I make repeated searches for things with "quotes" and whatnot on Google or on Stack Exchange sites. But for the most part, those challenges aren't particularly drawn…

It varies a lot based on what I'm doing. Sites that rely on ads like english-language¹ recipes or health information have a lot of "you're European so you're blocked altogether" or "let me check that the connection is secure , ah wait, here is a captcha for you to solve" pages. Anything that needs to do fraud detection usually hates me as well, perhaps because I have a phone number and bank account from another count…

Weird, I've not encountered region locks on recipe sites. From my experience it's mostly (smaller) news sites that do that.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#277
post #6

> A demo of AI Audit shared with TechCrunch showed how website owners can use the tool to see how AI models are scraping their sites. Cloudflare’s tool is able to see where each scraper that visits your site comes from, and offers selective windows to see how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting your site. And if I didn't authorize the freeloading copyright-launderi…

This is such a nice opportunity for 4chan weirdos to teach the AI some new slurs.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#278
post #33
post #11

Cloudflare found a new variation on their traditional service of protecting from abusers. This time, Cloudflare has formed a "marketplace" for the abuse from which they're protecting you, partnering with the abusers. And requiring you to use Cloudflare's service, or the abusers will just keep abusing you, without even a token payment. I'd need to ask the lawyer how close this is to technically being a protection rack…

> I'd need to ask the lawyer how close this is to technically being a protection racket, or other no-no. Wait 'til you find out how many of the DDoS-for-hire services that Cloudflare offers to protect you from are themselves protected by Cloudflare.

This comment demonstrates what an exceptional business it is - the house always wins.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#279

Earlier quoted context omitted.

Except it's some random company picking me for "extra checks" for no specified reason, and I don't even get a warning that there's this entity there.

There's literally a guard standing at the door. You are free to leave / not visit the site. And nobody owes you an explanation for the security practices of the business that you want to patronize.

This is after trying to enter, I can't retroactively opt out of the profiling and fingerprinting their captcha does for example.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#280

Earlier quoted context omitted.

Except it's some random company picking me for "extra checks" for no specified reason, and I don't even get a warning that there's this entity there.

There's literally a guard standing at the door. You are free to leave / not visit the site. And nobody owes you an explanation for the security practices of the business that you want to patronize.

Except that the guard in question is not under the business owner's control, and the business owner doesn't have a way to override the decision of the bouncer. In many cases, they don't even know how many customers the bouncer blocks.
Post reply on HN