Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

141–150 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#141

Earlier quoted context omitted.

There's a reason reddit started charging for API usage.

It surely wasn't to force users into their shitty app where they can't block ads and definitely had nothing to do with their IPO. It was the AI.

Ah yes, its only because of ONE singular reason they started charging for API usage. Are you okay? I'm listing one reason out of many as to why reddit started charging for API usage. After all, reddit is a for profit website.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#143

Earlier quoted context omitted.

They cant do anything other than bog down the internet. I havent found a single cf provided challenge I havent been able to get past in This is simply juat the first step in them implementing a marketplace and trying to get into LLM SEO. They dont care about your site or protecting it. They are gearing up to start making a cut in the Middle between scrapers and publishers. Why wouldnt I go DIRECTLY to the publisher a…

So what would you suggest as an alternative if I have a site where I don’t want the content used for LLM training?

Auth? Because whatever Cloudflare is doing isn't going to stop anyone serious about scraping data.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#144
post #112

Earlier quoted context omitted.

It's cloudflare and parasites like them that will make the internet un-free. It's already happening, I'm either blocked or back to 1998 load times be cause of "checking your browser". They are destroying the internet and will make it so only people who do approved things on approved browsers (meaning let advertising companies monetize their online activity) will get real access. Cloudflare isn't solving a problem, th…

Yep this terrifies me, 100%. We’re slowly losing the open internet and the frog is being boiled slowly enough that people are very happy to defend the rising temperature. If DDoS wasn’t a scary enough boogeyman to get people to install Cloudflare as a man-in-the-middle on all their website traffic, maybe the threat of AI scrapers will do the trick? The thing about this slow slide is it’s always defensible. Someone ca…

The Internet is not the first global network. Before the Internet, you had the global telephone network. It, too, strangulated end users, but eventually became stagnant, overpriced, and irrelevant. Super long-term, the current Internet is not immune from this. Internet standards are about getting as complicated and quirky as the old Bell stuff that was trying to make miles of buried copper the future, and if regulatory/commercial forces freeze this stuff in place, it's going to lead to stagnation eventually.

Something coming down the pike I think, for example, is that IPv4 addresses are going to get realllly expensive soon. That's going to lead to all sorts of interesting things in the Internet landscape and their applications.

I'm sure we'll probably have to spend some decades in the "approved devices and browers only" world before a next wave comes.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#145

Earlier quoted context omitted.

I don't think we should ban ad blockers, but I also think it's fair to suggest that the loss of organic traffic could be affecting the incentive to create new digital content, at least as much as the fear of having your content absorbed into an LLM's training data.

IMO the backlash against LLMs is more philosophical, a lot of people don’t like them or the idea of one learning from their content. Unless your website has some unique niche information unavailable anywhere else there’s no direct personal risk. RAG would be a more direct threat if anything.

It's not philosophical, it's economical.

AI scrapers increase traffic by maybe 10x (this varies per site) but provide no real value whatsoever to anyone. If you look at various forms of "value":

* Saying "this uses AI" might make numbers go up on the stock market if you manage to persuade people it will make numbers go up (see also: the market will remain irrational longer than you can remain solvent).

* Saying "this uses AI" might fulfill some corporate mandate.

* Asking AI to solve a problem (for which you would actually use the solution) allows you to "launder" the copyright of whatever source it is paraphrasing (it's well established that LLMs fail entirely if a question isn't found within their training set). Pirating it directly provides the same value, with significantly less errors/handholding.

* Asking AI to entertain you ... well, there's the novelty factor I guess, but even if people refuse to train themselves out of that obsession, the world is still far too full of options for any individual to explore them all. Even just the question of "what kind of ways can I throw some kind of ball around" has more answers than probably anyone here knows.

What am I missing?

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#146
post #112

Earlier quoted context omitted.

It's cloudflare and parasites like them that will make the internet un-free. It's already happening, I'm either blocked or back to 1998 load times be cause of "checking your browser". They are destroying the internet and will make it so only people who do approved things on approved browsers (meaning let advertising companies monetize their online activity) will get real access. Cloudflare isn't solving a problem, th…

Yep this terrifies me, 100%. We’re slowly losing the open internet and the frog is being boiled slowly enough that people are very happy to defend the rising temperature. If DDoS wasn’t a scary enough boogeyman to get people to install Cloudflare as a man-in-the-middle on all their website traffic, maybe the threat of AI scrapers will do the trick? The thing about this slow slide is it’s always defensible. Someone ca…

[dead]

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#147
post #137

Earlier quoted context omitted.

They put themselves as a middle man for almost the whole Internet, collect huge usage data about everyone and block anybody who doesn't use mainstream tools: https://news.ycombinator.com/item?id=42953508 https://news.ycombinator.com/item?id=13718752 https://news.ycombinator.com/item?id=23897705 https://news.ycombinator.com/item?id=41864632 https://news.ycombinator.com/item?id=42577076

valid. ...but OTOH it's their customers who want all of that and pay to get that, because the alternative is worse. rock and a hard place.

Right - do I want them getting some info from me, or do I want my IP address exposed?

Besides CloudFront, which still costs money, what other option is there for semi-privacy and caching for free?

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#148

Earlier quoted context omitted.

It surely wasn't to force users into their shitty app where they can't block ads and definitely had nothing to do with their IPO. It was the AI.

Ah yes, its only because of ONE singular reason they started charging for API usage. Are you okay? I'm listing one reason out of many as to why reddit started charging for API usage. After all, reddit is a for profit website.

> There's *A* reason

this u chief?

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#149
post #112

Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…

It's cloudflare and parasites like them that will make the internet un-free. It's already happening, I'm either blocked or back to 1998 load times be cause of "checking your browser". They are destroying the internet and will make it so only people who do approved things on approved browsers (meaning let advertising companies monetize their online activity) will get real access. Cloudflare isn't solving a problem, th…

Correction: extract monstreous profits. When I read about the revenues associated with Reddit AI deals, I can't even imagine what could possibly be deals that cover half of the internet. Cynically speaking, it's a genious level move.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#150
post #47

I’ve been using this for a while on my mastodon server and after a few tweaks to make sure it wasn’t blocking legit traffic it’s been really working great. Between Microsoft and Meta, they were hitting my services more than any other traffic combined which says a lot of you know how noisy mastodon can be. Server load went down dramatically. It also completely put a stop to perplexity as far as I can tell. And the rob…

This does nothing dude. Literally nothing. OpenAI or whoever are just going to hire people like me who dont get caught. Stop ruining the experience of users and allowing cf to fill the internet with more bloated javascript challenge pages and privacy invading fingerprinting. Stop making cf the police of the internet. We're literally handing the internet to this company on a silver platter to do MITM attacks on our pr…

Well the alternative is to not have an instance at all so… what do you suggest? I’m not paying for the other services, it’s already expensive enough to run the site.

The goal isn’t to stop 100% of scrapers, it was to reduce server load to a level that wasn’t killing the site.

Post reply on HN