Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

181–190 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#181
post #126

Earlier quoted context omitted.

Don’t put your stuff on the internet then, or put it behind a paywall/registration.

What you propose is making the web worse for everyone , instead of a minority of users (AI agents)

Huh? You have to login to Twit...er, X, Facebook, Insta, Snapchat, blah blah blah. After that, there's what 10% of the internet left. Seems like the open not-behind-paywall is the minority fo the interent

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#183
post #172
post #171

Earlier quoted context omitted.

"Security" in the real world doesn't get to profile people. Profiling is Cloudflare's entire business model.

>"Security" in the real world doesn't get to profile people 1. yes they do. have you ever been to vegas? there's cameras and facial recognition everywhere. outside of vegas, some bars and clubs also use ID scanning systems to enforce blacklists, and in most cases that system is outsourced to an external vendor. finally, ticketmaster requires an account to use, and to create an account you need to provide them your bi…

1. A blacklist isn't profiling. Known problem causing entities is entirely different than 'he looks suspicious', because the latter is often... Misused (to be polite).

2. Of course it is relevant. Because the more false positives they have the more money they can extort. They have negative incentive for their system to work properly.

P.S. ticketmaster is absolutely criminal, too.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#184
post #166

Earlier quoted context omitted.

I always spoil as many of these as possible. Sometimes it takes me a while to prove that I'm human, but I'm dead-set on convincing it that I'm a stupid human. Of course, I fantasize that some day a robo-car will crash because I taught it that there's really no difference between a motorcycle and a flight of stairs.

You'll just be lower on the list the AI makes of people that would be a threat.

I love this idea, some sort of inverse Roko's Basilisk. Tie a bunch of low-IQ data points to the sources a super AI is likely to first use to identify threats so as to eke out a few more days of existence.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#185

This seems like a gimmick. Isn't preventing crawling a sisyphean task? The only real difference this will make is further entrenching big players who have already crawled a ton of data. And if this feature comes at the cost of false positives and overbearing captchas, it will start to affect users.

My website contains millions of pages. It's not hard to notice the difference between a bot (or network) that wants to access all pages and a regular user.

Unless they are scraping it using residential botnet proxies, unique user-agents, unique device types, and etc...

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#186
post #173
post #144

Earlier quoted context omitted.

I mostly agree with you but do find it a fair point to suggest making it a straight-up paywall then. If they want some clients to pay for the content based on heuristic and black-box algorithms, that's going to be discriminatory, we just don't know to which groups (could be users from cheap connections or lower-income countries, could be unusual user agents like Ladybird on macOS, could be anything)

Perhaps, but I’m not sure how different that would be in practice. I have no more idea how the NYT implemented their paywall than Cloudflare does.

The scope of the average paywall is quite different, letting only some specific crawlers pass for indexing but not meaning to let anyone read who isn't subscribed. I can see the similarity you mean and it's an interesting case to compare with, but "everyone should pay, but we want to be findable" seems different to me from "only things that look like bots to us should pay". Perhaps also because the implementation of the former is easy (look up guidance for the search engines you want to be in; plain allowlist-based) and the latter is nigh impossible (needs heuristics and the bot operators can try to not match them but an average person can't do anything)

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#187

Boy I'm sick of clicking "Verify you are human" on everything from GitLab to banking apps running Cloudflare. Sick enough that I hope someone prominent at the EFF or similar takes Cloudflare to court over it. One company shouldn't be allowed to police access to the internet. And certainly shouldn't be allowed to start gatekeeping what is viewable by discriminating against the person or software doing the viewing. I w…

[deleted]

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#188
post #183
post #172

Earlier quoted context omitted.

>"Security" in the real world doesn't get to profile people 1. yes they do. have you ever been to vegas? there's cameras and facial recognition everywhere. outside of vegas, some bars and clubs also use ID scanning systems to enforce blacklists, and in most cases that system is outsourced to an external vendor. finally, ticketmaster requires an account to use, and to create an account you need to provide them your bi…

1. A blacklist isn't profiling. Known problem causing entities is entirely different than 'he looks suspicious', because the latter is often... Misused (to be polite). 2. Of course it is relevant. Because the more false positives they have the more money they can extort. They have negative incentive for their system to work properly. P.S. ticketmaster is absolutely criminal, too.

>2. Of course it is relevant. Because the more false positives they have the more money they can extort. They have negative incentive for their system to work properly.

What are the "false positives" in this context? It's specifically for blocking bots, and enrollment into the program to get unblocked is designed for bot owners. It's obviously not designed to extract money from regular users. I doubt there's even a straightforward way for regular users to pay to get unblocked via this channel. As the people who are running blocks and are blocked, I don't see what the issue is. Isn't it working as intended by definition?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#189
post #26

AI scrapers are parasites. I don't care whether you're OpenAI, Amazon, Meta, or some unknown startup. As soon as you generate a noticeable load on any of the servers I keep my eyes on, you'll get a blank 403 from all of the servers, permanently. I might allow a few select bots once there is clear evidence that they help bring revenue-generating visitors, like a major search engine does. Until then, if you want traini…

> AI scrapers are parasites.

I've been making crawlers for a living! Thanks for informing me that I'm a parasite.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#190
post #83

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

> gatekeep access to those who pay and those who don't, and that applied whether they are bots or people. I'm already constantly being classified as bot. Just today: To check if something is included in a subscription that we already pay for, I opened some product page on the Microsoft website this morning. Full-page error: "We are currently experiencing high demand. Please try again later." It's static content but i…

If you use Linux, the experience is terrible nowadays.

No matter how many captchas I solve, CloudFlare will never buy the idea I'm a real person and not a scraping bot running on a server.

I wonder if this kind of discrimination is even legal...

Post reply on HN