Cloudflare's new marketplace lets websites charge AI bots for scraping

Post by AyyEye » Mon, Sep 23, 2024, 6:28 PM UTC

gruez wrote: post #169

AyyEye wrote: post #164

Earlier quoted context omitted.

Because those websites cloudflare is performing racketeering-as-a-service for are open to the public.

Cloudflare isn't unilaterally inserting themselves between the website and you. They're contracted by the website owner to provide website security, just like how ticketmaster is contracted by the venue owner to provide ticketing. I don't see what the difference is.

Post by acdha » Mon, Sep 23, 2024, 6:35 PM UTC

Aachen wrote: post #144

acdha wrote: post #138

Earlier quoted context omitted.

So … it’s okay if they build their own system but you find it upsetting when they pay Cloudflare for a service?

I mostly agree with you but do find it a fair point to suggest making it a straight-up paywall then. If they want some clients to pay for the content based on heuristic and black-box algorithms, that's going to be discriminatory, we just don't know to which groups (could be users from cheap connections or lower-income countries, could be unusual user agents like Ladybird on macOS, could be anything)

Post by kylehotchkiss » Mon, Sep 23, 2024, 6:37 PM UTC

Post by dylan604 » Mon, Sep 23, 2024, 6:43 PM UTC

Aachen wrote: post #83

billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

> gatekeep access to those who pay and those who don't, and that applied whether they are bots or people. I'm already constantly being classified as bot. Just today: To check if something is included in a subscription that we already pay for, I opened some product page on the Microsoft website this morning. Full-page error: "We are currently experiencing high demand. Please try again later." It's static content but i…

Post by dylan604 » Mon, Sep 23, 2024, 6:45 PM UTC

klyrs wrote: post #166

diggan wrote: post #130

Earlier quoted context omitted.

> Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot Those ones are the fucking worst. I've noticed that if I try to succeed in these captchas too quickly, it'll just say "Sorry, try again" even when every click was correct, so instead, I've started going in slow motion and faking "misclicking" which makes it much more likely to accept me as human. I canno…

I always spoil as many of these as possible. Sometimes it takes me a while to prove that I'm human, but I'm dead-set on convincing it that I'm a stupid human. Of course, I fantasize that some day a robo-car will crash because I taught it that there's really no difference between a motorcycle and a flight of stairs.