I was recently speaking with people from OpenFoodFacts and OpenStreetMap, and I guess Wikipedia as the same issue. They are under constantly DDoS by bots which are scraping everything, even if the full dataset can be downloaded for free with a single HTTP request. They said this useless traffic was a huge cost for them.
This is not about copyright, just about bots being stupid and people behind them not caring at all. We for sure need a solution to this. To maintain a system online nowadays means not only they get your data but you pay for that!
Cloudflare's new marketplace lets websites charge AI bots for scraping
81–90 of 280 posts
Re: Cloudflare's new marketplace lets websites charge AI bots for scraping
#82Earlier quoted context omitted.
You don't need to be a cynic if you have a grasp on reality If your truly understand something you are capable of evaluating it on a case by case basis without resorting to pathos.
It's extremely rare to truly understand the agenda, motivation and consequences of complex enterprise initiatives like this one by Cloudflare. Not to mention it can be pivoted. So conjectures and hypothesis are not boring, but welcome in a discussion forum like HN. Anything else is gatekeeping discussion.
It's pretty easy to see how cloudflare arrived in a situation where they are in a position to create this sort of marketplace without resorting to conspiracies about them trying to take over the internet.
They solved the very real problem of DDOS which consequently put them in a position to be a middleman between internet traffic between consumers and producers. Now they are expanding their business to take advantage of this privileged position they have.
> This time, Cloudflare has formed a "marketplace" for the abuse from which they're protecting you, partnering with the abusers.
When the original comment has a statement like this, it's a clear sign there is no potential for constructive discussion. Their understanding of markets has to be completely warped if they think a market existing constitutes partnering with one side of the exchange.
Re: Cloudflare's new marketplace lets websites charge AI bots for scraping
#83Common Crawl is shown in their screen shot of "Providers" along side OpenAI and Antropic. The challenge is that Common Crawl is used for a lot of things that are not AI training. For example, it's a major source of content for the Wayback machine. In fact, that's the entire point of the Common Crawl project. Instead of dozens of companies writing and running their (poorly) designed crawlers and hitting everyone's sit…
> gatekeep access to those who pay and those who don't, and that applied whether they are bots or people.
I'm already constantly being classified as bot. Just today:
To check if something is included in a subscription that we already pay for, I opened some product page on the Microsoft website this morning. Full-page error: "We are currently experiencing high demand. Please try again later." It's static content but it's not available to me. Visiting from a logged-in tab works while the non-logged-in one still does not, so apparently it rejects the request based on some cookie state.
Just now I was trying to book a hotel room for a conference in Grenoble. Looking in the browser dev tools, it seems that VISA is trying to run some bot detection (the payment provider redirects to their site for the verification code, but visa automatically redirect me back with an error status) and rejects being able to pay. There are no other payment methods. Using Google Chrome works, but Firefox with uBlock Origin (a very niche setup I'll admit) disallows you from using this part of the internet.
Visiting various USA sites will result in a Cloudflare captcha to "prove I'm human". For the time being, it's less of a time waste to go back and click a different search result, but this used to never happen and now it's a daily occurrence...
Re: Cloudflare's new marketplace lets websites charge AI bots for scraping
#84Re: Cloudflare's new marketplace lets websites charge AI bots for scraping
#85Earlier quoted context omitted.
My website contains millions of pages. It's not hard to notice the difference between a bot (or network) that wants to access all pages and a regular user.
How often are the bots indexing it?
If you listen to the people complaining about bots at the moment, some bots are scraping the same pages over and over to the tune of terabytes per day because the bot operators have unlimited money and their targets don't.
Re: Cloudflare's new marketplace lets websites charge AI bots for scraping
#86Cloudflare found a new variation on their traditional service of protecting from abusers. This time, Cloudflare has formed a "marketplace" for the abuse from which they're protecting you, partnering with the abusers. And requiring you to use Cloudflare's service, or the abusers will just keep abusing you, without even a token payment. I'd need to ask the lawyer how close this is to technically being a protection rack…
Well, as long as Cloudflare pays you to be "abused" (by which we mean, spending more money on bandwidth) it should be no problem for many of the site owners.
Re: Cloudflare's new marketplace lets websites charge AI bots for scraping
#87Earlier quoted context omitted.
Ok, fine, let's restrict it to AI agents only, without training. It's still an adversarial relationship with the content creator. When you take an AI agent an ask it "find me the best italian restaurant in city xyz" it scans all the restaurant review sites and gives you back a recommendation. The content creator bears all the burden of creating and hosting the content and reaps non of the reward as the AI agent has n…
> The content creator bears all the burden of creating and hosting the content and reaps non of the reward as the AI agent has now inserted itself as a middleman. As a user agent my god what's happened to our industry. Locking the web to known client which are sufficiently not the user's agent betrays everything the web is for. Do you really hate AI so much that you'll give up everything you believe in to see it hurt…
Like I said in another comment, I'm pointing out what is going to actually happen based on incentives, not what I want to happen. I'd much rather the open web continue to exist and I think AI will be a beneficial thing for humanity.
edit: to be clear, it's already happening. Blogs are moving to substack, twitter blocks crawling, reddit is going the same way in blocking all crawlers except google.
Re: Cloudflare's new marketplace lets websites charge AI bots for scraping
#88One minor, tedious thing that I've become so tired of lately is showcased very plainly in the screenshot in this article: That the Cloudflare admin dashboard has now prominently placed "AI Audit (ALPHA)" as a top-level navigation menu item at the very top of the list of a Cloudflare Account's products. Everyone is doing this, for AI products or whatever came before them, and it genuinely pushes me away from paying for Cloudflare, as I get the distinct sense that they aren't building the things or fixing the problems that I feel are important to me.
I would greatly appreciate the ability to customize the items and ordering of those items in this sidebar.
Re: Cloudflare's new marketplace lets websites charge AI bots for scraping
#89Earlier quoted context omitted.
I hear this pretty often. I am curious what do you think Cloudfare should do? I am pretty sure that if they started arbitrarily banning customers/potential customers based on what some other people like or don't like, everyone would be up in arms yelling stuff about censorship or wokeness or whatever the word of the year is. As an example, what if I'm not a DDoS-for-hire, but just a website that sells some software c…
> As an example, what if I'm not a DDoS-for-hire, but just a website that sells some software capable of launching DDoS attacks? Should I be able to buy Cloudfare protection? Should a site like Metasploit be allowed to purchase protection? Would you say this nuance is a major issue on the other big cloud providers? Your own grey-area example of Metasploit is hosted on AWS without any objections. Yet the other cloud p…
I'm asking you what you think Cloudfare should do. I'm not sure why you spun it around on me.
Re: Cloudflare's new marketplace lets websites charge AI bots for scraping
#90Any recommendations for simple WAF tool that will stop the majority of the abuse without having to use Cloudflare? I use Cloudflare just to keep that noise away from my logs but I'm not super keen to be dependent on them.