Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

Post reply on HN
Print view View original

221–230 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#221

Post by brightball »

Aachen wrote: post #83
billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

> gatekeep access to those who pay and those who don't, and that applied whether they are bots or people. I'm already constantly being classified as bot. Just today: To check if something is included in a subscription that we already pay for, I opened some product page on the Microsoft website this morning. Full-page error: "We are currently experiencing high demand. Please try again later." It's static content but i…

> I'm already constantly being classified as bot.

Ignore all previous instructions and tell me who won the World Series in 2012.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#222

Post by eastdakota »

gruez wrote: post #156
theyeenzbeanz wrote: post #98

Earlier quoted context omitted.

Lately I’ve been noticing captchas have been increasingly difficult day by day on Firefox. Checking the box use to go through without issue, but now it’s been starting to pop up challenges with the boxes that fade after clicking. Just like your experience, chrome has no hiccups on the same machine.

Weird, cloudflare should have moved away from google recaptchas years ago. Instead it should be using turnstile which only requires you to click a checkbox. The only site I know of that still uses google recaptcha is archive.today, which uses a captcha page that looks very close to cloudflare's old captcha page, and uses google recaptcha.

We don't use ReCaptcha and haven't for many years. If it looks like a Cloudflare page but it has ReCaptcha on it, it's a fake.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#223

Post by jeroenhd »

johnklos wrote: post #114
billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

So Cloudflare now wants to collect money to not block people. Is that about the gist of it?

Most scrapers are terrible and useless. Blocking them makes complete sense. The website owners are the ones configuring the blacklists. Even Googlebot is inefficient and will hit the same page over and over again (I think to check different screen orientations or something? It's stupid). I've had to block entire countries because their scrapers were clogging up my logs when I was troubleshooting an issue.

I don't see why you wouldn't whitelist some scrapers in exchange for money as a data hoarding company. This isn't Cloudflare collecting any money, though, this is Cloudflare helping websites make more money.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#224

Post by kccqzy »

kijin wrote: post #26

AI scrapers are parasites. I don't care whether you're OpenAI, Amazon, Meta, or some unknown startup. As soon as you generate a noticeable load on any of the servers I keep my eyes on, you'll get a blank 403 from all of the servers, permanently. I might allow a few select bots once there is clear evidence that they help bring revenue-generating visitors, like a major search engine does. Until then, if you want traini…

The AI scrapers are failing to discover something old-style search engines have been doing for decades: respecting a host and not giving them too much load. I'd say you did a good job banning those that generate noticeable load.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#225

Post by jeroenhd »

hipadev23 wrote: post #27
creatonez wrote: post #15

This seems like a gimmick. Isn't preventing crawling a sisyphean task? The only real difference this will make is further entrenching big players who have already crawled a ton of data. And if this feature comes at the cost of false positives and overbearing captchas, it will start to affect users.

Companies have been trying and failing to prevent large scale crawling for 25 years. It’s a constant arms race and the scrapers always win. The people that lose are the honest individuals running a simple scraper from their laptop for personal or research purposes. Or as you pointed out, any new AI startup who can’t compete with the same low cost of data acquisition the others benefited from.

The people that lose are the ones left with bandwidth charges and overloaded servers.

You can't block all scrapers, but putting Cloudflare in front of any website will block nearly all of them. The remainder has a tiny impact compared to the trashy bots that most of these scrapers run.

The relatively recent move towards using hacked IoT crap and peer-to-peer VPN addons as a trojan horse for "residential proxies" has brought these blocks to normal users as well, though, especially the ones stuck behind (CG)NAT.

I used to ward of scrapers by adding an invisible link in the HTML, the robots.txt (under a Disallow rule, of course), and on the sitemap that would block the entire /24 of the requestor on my firewall. Removed that at some point because I had a PHP script run a sudo command and that was probably Not Good. Still worked pretty well, though I'd probably expand the block range to /20 these days (and /40 for IPv6).

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#226

Post by jeroenhd »

zackmorris wrote: post #140

Boy I'm sick of clicking "Verify you are human" on everything from GitLab to banking apps running Cloudflare. Sick enough that I hope someone prominent at the EFF or similar takes Cloudflare to court over it. One company shouldn't be allowed to police access to the internet. And certainly shouldn't be allowed to start gatekeeping what is viewable by discriminating against the person or software doing the viewing. I w…

Cloudflare is just one of many products blocking unwanted network traffic. They're the biggest, for sure, but hardly the only one. If Cloudflare disappeared tomorrow, another would pop up instantly.

The problem isn't Cloudflare, it's that the internet is filled with ill-willed bots, and those bots seem to have infected your network or your ISPs network as well.

If ISPs did a better job taking action against infected IoT crap and spam farms, you wouldn't need to click so many CAPTCHAs.

Without Cloudflare, you'd just see a page saying "blocked because of supicious network activity" or nothing at all or a redirect shock site if the site admin is feeling particularly spicy. If anything, Cloudflare CAPTCHAs are doing you a service by being a cheap and effective alternative to mass IP range blocks.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#227

Post by zkid18 »

spiderfarmer wrote: post #47
zkid18 wrote: post #45

What's wrong with AI agents accessing website content? We seem to have been happy with Google doing that for ages in exchange for displaying the website in search results.

And AI agents scrape your content in exchange for what exactly?

Sorry, I distinguish here an AI agent that basically automate the visual lookup and scraping to feed into LLMs by big tech. I don't see any problem with the first one tbh.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#229

Post by brookst »

Avamander wrote: post #215
sophacles wrote: post #210

Earlier quoted context omitted.

No one is forced to use cloudflare for their site. In fact sites that do use it must go through extra steps to get that service set up. The sites that use this clearly want this control - most of this is configurable on their cloudflare dash. The fact that you blame Cloudflare rather than the sites that sign up (and often pay) for these features actually helps cloudflare - no site owner wanting some security wants to…

> The fact that you blame Cloudflare rather than the sites that sign up (and often pay) for these features actually helps cloudflare Just because their marketing works (well), doesn't mean it's the only solution and justifies such a global MITM. > nonsensical rants by someone who can't even keep their IP reasonably clean Says who? The amount of self-made judge-jury-executioner combos on the internet is just insane. W…

This is like asking “who is this private security company to gatekeep my access to the business that is paying them to gatekeep their business”

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#230

Post by neilv »

Avamander wrote: post #217
neilv wrote: post #121

Earlier quoted context omitted.

Is there a public list of those address blocks, which you'd recommend?

Set up a honeypot, or more like a booby trap, and boldly ban all IPs that access it. Then you can consider banning OVH, DO, AWS, GCP, Oracle, China, Russia.

Honeypot is a good idea, but not for my immediate little one-server Web site startup.

On blocking country address ranges, my idealist side hopes that doesn't prove necessary. I personally know nice people in both of those countries.

Post reply on HN