Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

Post reply on HN
Print view View original

151–160 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#151

Post by jkestner »

diggan wrote: post #130
Aachen wrote: post #102

Earlier quoted context omitted.

Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot and this is the highest difficulty level to prove your lack of guilt. I get these only when using a browser that isn't already full of advertising cookies (edit: which, to be clear, I hope is still considered an acceptable state to have your browser in)

> Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot Those ones are the fucking worst. I've noticed that if I try to succeed in these captchas too quickly, it'll just say "Sorry, try again" even when every click was correct, so instead, I've started going in slow motion and faking "misclicking" which makes it much more likely to accept me as human. I canno…

Harrison Bergeron but for AI

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#152

Post by internetter »

wpm wrote: post #126
acdha wrote: post #123

Earlier quoted context omitted.

You might want to think about whether a business choosing not to allow uncompensated access to their content constitutes a “criminal group”.

Don’t put your stuff on the internet then, or put it behind a paywall/registration.

What you propose is making the web worse for everyone, instead of a minority of users (AI agents)

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#153

Post by epc »

neilv wrote: post #121
epc wrote: post #107

Earlier quoted context omitted.

I’ve just taken to blocking entire swaths of cloud services IP networks. I don’t care what the intentions are, my personal sites don’t get the infinite bandwidth to put up with a thousands of poorly written spiders.

Is there a public list of those address blocks, which you'd recommend?

Not that I know of, but each service seems to publish a list (some in text, some JSON). I’ll reply later with the URLs of the ones I have.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#154

Post by [deleted] »

wpm wrote: post #126
acdha wrote: post #123

Earlier quoted context omitted.

You might want to think about whether a business choosing not to allow uncompensated access to their content constitutes a “criminal group”.

Don’t put your stuff on the internet then, or put it behind a paywall/registration.

[deleted]

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#155

Post by AlienRobot »

billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

I think this is a temporary problem. In a few years many AI companies will run out of VC money, others will be only after "low-background" content made before AI spam. Maybe one day nature will heal.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#156

Post by gruez »

theyeenzbeanz wrote: post #98
Aachen wrote: post #83

Earlier quoted context omitted.

> gatekeep access to those who pay and those who don't, and that applied whether they are bots or people. I'm already constantly being classified as bot. Just today: To check if something is included in a subscription that we already pay for, I opened some product page on the Microsoft website this morning. Full-page error: "We are currently experiencing high demand. Please try again later." It's static content but i…

Lately I’ve been noticing captchas have been increasingly difficult day by day on Firefox. Checking the box use to go through without issue, but now it’s been starting to pop up challenges with the boxes that fade after clicking. Just like your experience, chrome has no hiccups on the same machine.

Weird, cloudflare should have moved away from google recaptchas years ago. Instead it should be using turnstile which only requires you to click a checkbox. The only site I know of that still uses google recaptcha is archive.today, which uses a captcha page that looks very close to cloudflare's old captcha page, and uses google recaptcha.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#157

Post by zebomon »

Here's a look at my AI Audit on Bingeclock for anyone who's curious. Interesting drop in the last 48 hours given that it coincided with Cloudflare's announcement.

https://www.bingeclock.com/blog/img/ai-audit-cloudflare-0923...

The payment program sounds intriguing, I suppose. I can't imagine it will do much to move the needle for websites that will become unviable due to traffic drain. Without a doubt, AI scrapers will (quite rationally from their POV) avoid anything but nominal payments until they're forced to do otherwise.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#158

Post by gruez »

Mistletoe wrote: post #115
johnklos wrote: post #114

Earlier quoted context omitted.

So Cloudflare now wants to collect money to not block people. Is that about the gist of it?

> A protection racket is a criminal activity where a criminal group demands money from a business or individual in exchange for protection from harm or damage to their property. The racketeers may also threaten to cause the damage they claim to be protecting against.

How is this different than say, ticketmaster charging money to not get "blocked" from a venue (ie. a ticket)?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#159

Post by Icathian »

zackmorris wrote: post #140

Boy I'm sick of clicking "Verify you are human" on everything from GitLab to banking apps running Cloudflare. Sick enough that I hope someone prominent at the EFF or similar takes Cloudflare to court over it. One company shouldn't be allowed to police access to the internet. And certainly shouldn't be allowed to start gatekeeping what is viewable by discriminating against the person or software doing the viewing. I w…

Do you also get mad at companies that make locks when people install them on their front doors?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#160

Post by mqus »

diggan wrote: post #130
Aachen wrote: post #102

Earlier quoted context omitted.

Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot and this is the highest difficulty level to prove your lack of guilt. I get these only when using a browser that isn't already full of advertising cookies (edit: which, to be clear, I hope is still considered an acceptable state to have your browser in)

> Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot Those ones are the fucking worst. I've noticed that if I try to succeed in these captchas too quickly, it'll just say "Sorry, try again" even when every click was correct, so instead, I've started going in slow motion and faking "misclicking" which makes it much more likely to accept me as human. I canno…

As soon as I notice that I got this slow-fade-captcha, I will intentionally click all the wrong fields until I get a reasonable captcha. Not sure this makes a difference but it kinda works
Post reply on HN