Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

Post reply on HN
Print view View original

111–120 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#111

Post by Aachen »

esperent wrote: post #96
Aachen wrote: post #83

Earlier quoted context omitted.

> gatekeep access to those who pay and those who don't, and that applied whether they are bots or people. I'm already constantly being classified as bot. Just today: To check if something is included in a subscription that we already pay for, I opened some product page on the Microsoft website this morning. Full-page error: "We are currently experiencing high demand. Please try again later." It's static content but i…

> We are currently experiencing high demand. Please try again later. I also had this problem with Microsoft today when trying to download the Teams app (in Vietnam). We use MS Teams at work and onboard one or two people a week. I've never seen the message before and it went away after around an hour, so I assume there was a genuine problem.

Perhaps, but it loaded fine in Chrome as well as a logged-in tab. It only rejected the Firefox no-cookies user agent. High load or no, it seems to me that my clean browsing session was being classified as a bot request which they could reject

Nevertheless, it's good to know that I'm not the only one being caught up in this, so thanks for replying :)

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#112

Post by ajsnigrutin »

Aachen wrote: post #83
billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

> gatekeep access to those who pay and those who don't, and that applied whether they are bots or people. I'm already constantly being classified as bot. Just today: To check if something is included in a subscription that we already pay for, I opened some product page on the Microsoft website this morning. Full-page error: "We are currently experiencing high demand. Please try again later." It's static content but i…

Same here... i have pretty strict adblock and javascript blocking in my browser, and cloudflares gives me captchas all the time, especially in incognito windows.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#113

Post by ajsnigrutin »

Aachen wrote: post #102
theyeenzbeanz wrote: post #98

Earlier quoted context omitted.

Lately I’ve been noticing captchas have been increasingly difficult day by day on Firefox. Checking the box use to go through without issue, but now it’s been starting to pop up challenges with the boxes that fade after clicking. Just like your experience, chrome has no hiccups on the same machine.

Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot and this is the highest difficulty level to prove your lack of guilt. I get these only when using a browser that isn't already full of advertising cookies (edit: which, to be clear, I hope is still considered an acceptable state to have your browser in)

Aw man, you haven't seen the 'captchas' of arkose labs yet... those are a pain (twitter used to have them some time ago).

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#114

Post by johnklos »

billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

So Cloudflare now wants to collect money to not block people. Is that about the gist of it?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#115

Post by Mistletoe »

johnklos wrote: post #114
billyhoffman wrote: post #28

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

So Cloudflare now wants to collect money to not block people. Is that about the gist of it?

> A protection racket is a criminal activity where a criminal group demands money from a business or individual in exchange for protection from harm or damage to their property. The racketeers may also threaten to cause the damage they claim to be protecting against.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#116

Post by Aachen »

ajsnigrutin wrote: post #113
Aachen wrote: post #102

Earlier quoted context omitted.

Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot and this is the highest difficulty level to prove your lack of guilt. I get these only when using a browser that isn't already full of advertising cookies (edit: which, to be clear, I hope is still considered an acceptable state to have your browser in)

Aw man, you haven't seen the 'captchas' of arkose labs yet... those are a pain (twitter used to have them some time ago).

Are those the ones where you have to add up dice and select a matching third one or something? The ones GitHub used for registration, say, ~9 months ago?

You're right! I forgot about those. A colleague and I tried to complete it independently but literally could not. One run would take multiple minutes and on the second try I was more diligent (taking even longer) and certain I did all the math correctly, but registration was still being rejected. Our new colleague did not sign up for GitHub that day and got the repository from a colleague who already had access instead

Edit: seems that's yet another one. Arkose https://www.arkoselabs.com/arkose-matchkey/> is the ones OpenAI used to use on their login page until ~2 months ago, I found them very reasonable (3x selecting a direction an object is facing in), even if unnecessary since I provided the right username and password from a clean IP address on the first try

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#117

Post by Aachen »

ajsnigrutin wrote: post #112
Aachen wrote: post #83

Earlier quoted context omitted.

> gatekeep access to those who pay and those who don't, and that applied whether they are bots or people. I'm already constantly being classified as bot. Just today: To check if something is included in a subscription that we already pay for, I opened some product page on the Microsoft website this morning. Full-page error: "We are currently experiencing high demand. Please try again later." It's static content but i…

Same here... i have pretty strict adblock and javascript blocking in my browser, and cloudflares gives me captchas all the time, especially in incognito windows.

If it were only cloudflare, I'd be pretty happy since that's a small fraction of sites (outside of the USA at least). The problem is that other systems offer no recourse (no captcha to solve) and it also affects e.g. being able to pay for stuff. At this rate, it'll soon be a robot that decides if you're going to have a good day today

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#118

Post by esperent »

Aachen wrote: post #111
esperent wrote: post #96

Earlier quoted context omitted.

> We are currently experiencing high demand. Please try again later. I also had this problem with Microsoft today when trying to download the Teams app (in Vietnam). We use MS Teams at work and onboard one or two people a week. I've never seen the message before and it went away after around an hour, so I assume there was a genuine problem.

Perhaps, but it loaded fine in Chrome as well as a logged-in tab. It only rejected the Firefox no-cookies user agent. High load or no, it seems to me that my clean browsing session was being classified as a bot request which they could reject Nevertheless, it's good to know that I'm not the only one being caught up in this, so thanks for replying :)

No worries. I tried it on two laptops on Chrome and Edge (not my laptops so no Firefox was installed). Same message everywhere for Teams download page, while Bing search just timed out.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#119

Post by addaon »

Aachen wrote: post #103
immibis wrote: post #85

Earlier quoted context omitted.

If you listen to the people complaining about bots at the moment, some bots are scraping the same pages over and over to the tune of terabytes per day because the bot operators have unlimited money and their targets don't.

> because the bot operators have unlimited money I rather think the cause is that inbound bandwidth is usually free, so they need maybe 1/100th of the money because requests are smaller than responses (plus discounts they get for being big customers)

> I rather think the cause is that inbound bandwidth is usually free, so they need maybe 1/100th of the money because requests are smaller than responses (plus discounts they get for being big customers)

Seems like there's the potential to take advantage of this for a semi-custom protocol, if there's a desire to balance costs for serving data while still making things available to end users. We'd have the server reply to the initial request with a new HTTP response instructing the client to re-request with a POST containing an N-byte (N = data size) one-time pad. The client can receive this, generate random data (or all zeros, up to the client); and the server then will send the actual response XOR'd with the one-time pad.

Upside: Most end users don't pay for upload; if bot operators do, this incurs a dollar cost only to them. Downside: Increased download cost for the web site operator (but we've postulated that this is small compared to upload cost), extra round trip, extra time for each request (especially for end users with asymmetric bandwidth).

Eh, just a thought.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#120

Post by jsheard »

sharpshadow wrote: post #109

It is indeed a huge waste to scrape the same whole site for changes and new content. If Cloudflare is capable to maintain an overview about changes and updates it could save a lot of resources. The site could tell cloudflare directly what changed and cloudflare could tell the AI. The AI buys the changes and cloudflare pays the site keeps a margin.

The sitemap.xml spec already has fields for indicating the last time a page was changed and how often it's expected to change in the future, so that search engines can optimize their updates accordingly, but AI scrapers tend to disregard that and just download the same unchanged page 10,000 times for the hell of it.
Post reply on HN