Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

201–210 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#201

Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…

> This feels like a step down the path to a world where the majority of websites use sophisticated security products that gatekeep access to those who pay and those who don't ... and that future has been a long time coming. People who want an alternative to advertising-supported online content? This is what that alternative looks like. Very few content providers are going to roll their own infrastructure to standardi…

> People who want an alternative to advertising-supported online content? This is what that alternative looks like.

Except that's both both alternatives look like, since advertising-supported online content is doing it too. Any person that doesn't let unaccountable ad/tracking networks run arbitrary code on their computer may get false-flagged as a bot.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#202

Earlier quoted context omitted.

It's acceptable, but suspicious. Two standard deviations away from the median browser (and a lot more like the configuration of a scraper, which would get reloaded in some Docker instance frequently with a fresh empty cookie jar because storing data costs infrastructure).

You mean Edge? Chrome stands a 65.2% ( 1 deviation ) Safari at 18.57% ( 2 deviations ), so Edge at 5.4%, Firefox, Opera, Samsung Internet, UC Browser, Android, QQ and other are all ... deviants? https://gs.statcounter.com/browser-market-share I use Firefox nightly which does not even show up statistically...

Not sure if they're using user agent. Probably not because it's so easy to forge UA.

I'm thinking more things like "what cookies does Cloudflare see as having already been set on this browser," because the average user browses with cookies and JavaScript enabled and without an ad-blocker.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#203
post #158

Earlier quoted context omitted.

How is this different than say, ticketmaster charging money to not get "blocked" from a venue (ie. a ticket)?

It isn't. Ticketmaster is also a way to dominant middleman with way too much influence in the sector.

"cloudflare is engaging in monopolistic behavior" would be the saner take here, but the OP was specifically accusing cloudflare of being a "protection racket". Ticketmaster might be engaging in illegal monopolistic behavior in the ticket space, but nobody seriously thinks they're engaging in a "protection racket" over access to venues.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#204
post #193

Is anybody else seeing an absolutely massive amount of Amazonbot crawls on their site? What are they up to? And why so aggressively?

Most likely aspiring AI startups gathering as much data as they can before regulation jaws snap shut around them cutting off the blood stream. In this AI race(hype), data is finally the ultimate gold. Also at the rate the information is polluted by GenAI junk all over, any remnants of real data is holy grail.

So any unknown or upcoming AIs would just show as Amazon?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#206

Earlier quoted context omitted.

If you use Linux, the experience is terrible nowadays. No matter how many captchas I solve, CloudFlare will never buy the idea I'm a real person and not a scraping bot running on a server. I wonder if this kind of discrimination is even legal...

Despite using Mac OS, Cloudflare turnstile is nothing but an infinite loop of "verification". I am using Firefox with basic privacy protections enabled. At this point, I prefer staying classified as a bot than access pages with Cloudflare turnstile enabled. Before infinite loops from Cloudflare, I had noticed that Google Captcha on Firefox would frequently reject audio challenges and require a lot more work than othe…

Same. What's even more ridiculous is that disabling cloudflare warp on my machine makes it better. Cloudflare doesn't even trust Cloudflare.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#207
post #153
post #121

Earlier quoted context omitted.

Is there a public list of those address blocks, which you'd recommend?

Not that I know of, but each service seems to publish a list (some in text, some JSON). I’ll reply later with the URLs of the ones I have.

This is what I have, see another reply for shared IP lists:

  https://ip-ranges.amazonaws.com/ip-ranges.json

  https://www.digitalocean.com/geo/google.csv
  
  https://www.gstatic.com/ipranges/cloud.json

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#208
post #153
post #121

Earlier quoted context omitted.

Is there a public list of those address blocks, which you'd recommend?

Not that I know of, but each service seems to publish a list (some in text, some JSON). I’ll reply later with the URLs of the ones I have.

I also found this but haven't validated it yet: https://github.com/femueller/cloud-ip-ranges

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#209
post #198
post #191

Earlier quoted context omitted.

> It's specifically for blocking bots Define "bots" in a way computers can understand. > What are the "false positives" in this context? Regular users that cloudflare (profiles) accuses of being bots. God help you if you want to block trackers or something else that's not regular. > I doubt there's even a straightforward way for regular users to pay to get unblocked via this channel This is part of the problem. But h…

>Define "bots" in a way computers can understand. How is having a specific definition relevant to this conversation? An approximate definition of "a human using a browser to visit a site" probably suffices, without having to get into weird edge cases like "but what if they programmed lynx to visit your site at 3am when they're asleep?". >Regular users that cloudflare (profiles) accuses of being bots. God help you if…

> How is having a specific definition relevant to this conversation?

Because it's a computer that automatically does it. That's the entire problem here. Humans are not in the loop, except collecting the paychecks.

> An approximate definition of "a human using a browser to visit a site" probably suffices

Humans are not doing the blocking. "Approximate" is not good enough when, for example, I need to go to a coffee shop and use an entirely different computer to trick cloudflare into letting me order from my longtime vendor. And I must repeat that my work computer is doing absolutely nothing interesting. My job and livelihood depend on this.

> without having to get into weird edge cases like "but what if they programmed lynx to visit your site at 3am when they're asleep?".

What about an edge case like 'using your bone stock phone to visit a site once'?

What about all the poor suckers that installed an app that loaded legal software designed specifically to use their phone's connection for scraping a la brightdata? Residential proxies are big business.

There are billions of users on the web. It is one gigantic pile of edge cases. And that's entirely the point. CF may get some right but they also get plenty wrong with no recourse (but now you may be allowed to pay them money for access).

> So they're damned if they do, damned if they do?

Yes. Their entire business model is "we have a magic crystal ball that only stops 'the wrong people'™ from your website".

> God forbid that site operators have agency over what visitors they allow on their sites!

They quite literally don't have that agency. This goes back to "define bot". There are zero websites that would want to block me from making purchases from them and yet that is exactly the result in the end. I had to change vendors for a five figure order because I was up against a deadline and couldn't get around the cloudflare block from my office, and the vendor had closed for the night so I couldn't call them and bypass the whole mess.

Afterwards we spent nearly a week trying to figure out how to let me buy from them again and they were willing to keep going back and forth with CF on my behalf but I was over it and not going to spend any more time. Now I'm using the non-CF vendor to their disappointment. So much for agency.

> I use ublock, resistfingerpnting, and a VPN. That probably puts me in the 95+ percentile in terms of suspiciousness. Yet the most hassle I get from cloudflare is the turnstile challenges can be solved by clicking a checkbox.

Good for you? I have a bone-stock computer on its own connection just to try to work around this BS and yet I still sometimes get an infinite loop where the checkbox never goes away.

When I have my VPN to our euro office on I am 100% unable to access CF sites whatsoever. Been that way for as long as I can remember.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#210
post #161

Earlier quoted context omitted.

So Cloudflare now wants to collect money to not block people. Is that about the gist of it?

It really is a fantastic scam. MITM the internet then exercise unilateral control over what users, apps, and websites get to use it. Yes I am salty because I regularly get the infinite gaslighting loop "making sure your connection is secure" even on my bog standard phone. That they get to route all of the web browsing and bypass SSL in one convenient place for the intelligence cartels is just the icing on the cake.

No one is forced to use cloudflare for their site. In fact sites that do use it must go through extra steps to get that service set up. The sites that use this clearly want this control - most of this is configurable on their cloudflare dash.

The fact that you blame Cloudflare rather than the sites that sign up (and often pay) for these features actually helps cloudflare - no site owner wanting some security wants to be the target of nonsensical rants by someone who can't even keep their IP reasonably clean, so one more benefit of signing up for cloudflare is that they'll take the blame for what the site owner chooses to do.

Post reply on HN