Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

Post reply on HN
Print view View original

161–170 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#161

Post by AyyEye »

johnklos wrote: post #114
billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

So Cloudflare now wants to collect money to not block people. Is that about the gist of it?

It really is a fantastic scam. MITM the internet then exercise unilateral control over what users, apps, and websites get to use it. Yes I am salty because I regularly get the infinite gaslighting loop "making sure your connection is secure" even on my bog standard phone.

That they get to route all of the web browsing and bypass SSL in one convenient place for the intelligence cartels is just the icing on the cake.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#163

Post by gruez »

zackmorris wrote: post #140

Boy I'm sick of clicking "Verify you are human" on everything from GitLab to banking apps running Cloudflare. Sick enough that I hope someone prominent at the EFF or similar takes Cloudflare to court over it. One company shouldn't be allowed to police access to the internet. And certainly shouldn't be allowed to start gatekeeping what is viewable by discriminating against the person or software doing the viewing. I w…

>Sick enough that I hope someone prominent at the EFF or similar takes Cloudflare to court over it.

On what basis? It sucks that you can't visit those sites without going through an interstitial, but at the end of the day, those sites are essentially private property and the owners can impose whatever requirements they want on visitors. It's not any different than sites that have registration walls, for instance.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#164

Post by AyyEye »

gruez wrote: post #158
Mistletoe wrote: post #115

Earlier quoted context omitted.

> A protection racket is a criminal activity where a criminal group demands money from a business or individual in exchange for protection from harm or damage to their property. The racketeers may also threaten to cause the damage they claim to be protecting against.

How is this different than say, ticketmaster charging money to not get "blocked" from a venue (ie. a ticket)?

Because those websites cloudflare is performing racketeering-as-a-service for are open to the public.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#165

Post by lolinder »

Aachen wrote: post #95
lolinder wrote: post #46

Earlier quoted context omitted.

Either AI training is fair use or it isn't. If it's fair use then businesses shouldn't get a say in whether the data can be used for it. If it isn't, then the answer to your question is copyright law. Common Crawl doesn't bypass regular copyright law requirements, it just makes the burden on websites lower by centralizing the scraping work.

Copyright is only part of the equation, there's also the use of other people's resources If what a government receptionist says is copyright-free, you still can't walk into their office thousands of times per day and ask various questions to learn what human answers are like in order to train your artificial neural network The amount of scraping that happened in ~2020 as compared to 2024 is orders of magnitude differ…

But here we're talking about Common Crawl being included in this scheme, which is explicitly designed to make it easier to use them than to make your own bad robot.

You block Common Crawl and all you'll be left with is the abusive bots that find workarounds.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#166

Post by klyrs »

diggan wrote: post #130
Aachen wrote: post #102

Earlier quoted context omitted.

Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot and this is the highest difficulty level to prove your lack of guilt. I get these only when using a browser that isn't already full of advertising cookies (edit: which, to be clear, I hope is still considered an acceptable state to have your browser in)

> Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot Those ones are the fucking worst. I've noticed that if I try to succeed in these captchas too quickly, it'll just say "Sorry, try again" even when every click was correct, so instead, I've started going in slow motion and faking "misclicking" which makes it much more likely to accept me as human. I canno…

I always spoil as many of these as possible. Sometimes it takes me a while to prove that I'm human, but I'm dead-set on convincing it that I'm a stupid human. Of course, I fantasize that some day a robo-car will crash because I taught it that there's really no difference between a motorcycle and a flight of stairs.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#167

Post by gruez »

laserbeam wrote: post #148
zackmorris wrote: post #140

Boy I'm sick of clicking "Verify you are human" on everything from GitLab to banking apps running Cloudflare. Sick enough that I hope someone prominent at the EFF or similar takes Cloudflare to court over it. One company shouldn't be allowed to police access to the internet. And certainly shouldn't be allowed to start gatekeeping what is viewable by discriminating against the person or software doing the viewing. I w…

Something I never considered, I wonder how clicking to be a human works for people with disabilities. There’s gotta be accessibility features there, and I bet bots are abusing them.

At least for cloudflare "captchas", you don't have no solve anything, only click a button. Therefore it's pretty accessible. My guess is that they care less about whether you're a human or not, and more about imposing resource costs on any attacker, because solving those challenges requires a full browser runtime (ie. hundreds of megs of memory + some non-trivial amount of CPU time). That's significantly more expensive than you spamming requests.post() with on a thousand threads.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#168

Post by Wingman4l7 »

laserbeam wrote: post #148
zackmorris wrote: post #140

Boy I'm sick of clicking "Verify you are human" on everything from GitLab to banking apps running Cloudflare. Sick enough that I hope someone prominent at the EFF or similar takes Cloudflare to court over it. One company shouldn't be allowed to police access to the internet. And certainly shouldn't be allowed to start gatekeeping what is viewable by discriminating against the person or software doing the viewing. I w…

Something I never considered, I wonder how clicking to be a human works for people with disabilities. There’s gotta be accessibility features there, and I bet bots are abusing them.

Or, the company leaves the accessibility alternative broken, and shrugs.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#169

Post by gruez »

AyyEye wrote: post #164
gruez wrote: post #158

Earlier quoted context omitted.

How is this different than say, ticketmaster charging money to not get "blocked" from a venue (ie. a ticket)?

Because those websites cloudflare is performing racketeering-as-a-service for are open to the public.

Cloudflare isn't unilaterally inserting themselves between the website and you. They're contracted by the website owner to provide website security, just like how ticketmaster is contracted by the venue owner to provide ticketing. I don't see what the difference is.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#170

Post by shadowgovt »

zackmorris wrote: post #140

Boy I'm sick of clicking "Verify you are human" on everything from GitLab to banking apps running Cloudflare. Sick enough that I hope someone prominent at the EFF or similar takes Cloudflare to court over it. One company shouldn't be allowed to police access to the internet. And certainly shouldn't be allowed to start gatekeeping what is viewable by discriminating against the person or software doing the viewing. I w…

> I worry that Cloudflare will keep escalating this unless they're sent a strong signal that it's not supported by the tech community.

I don't think that it's not supported by the tech community. Much of that community is on the receiving end of the bad actors. I know that depending on the day I, for one, have muttered under my breath "This would be much easier if everyone were using the same damn web browser."

Post reply on HN