Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

nytimes.com

Post reply on HN
Print view View original

251–260 of 342 posts

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#251

Post by robrenaud »

o11c wrote: post #145
Boldened15 wrote: post #68

Earlier quoted context omitted.

IMO the backlash against LLMs is more philosophical, a lot of people don’t like them or the idea of one learning from their content. Unless your website has some unique niche information unavailable anywhere else there’s no direct personal risk. RAG would be a more direct threat if anything.

It's not philosophical, it's economical. AI scrapers increase traffic by maybe 10x (this varies per site) but provide no real value whatsoever to anyone . If you look at various forms of "value": * Saying "this uses AI" might make numbers go up on the stock market if you manage to persuade people it will make numbers go up (see also: the market will remain irrational longer than you can remain solvent). * Saying "thi…

Why are 100s of millions of people using AI if it is providing no value?

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#252

Post by visarga »

abalashov wrote: post #101

Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…

> everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop

I think on the contrary, who sets the prompts stands to get benefits, the AI provider gets a flat fee, and authors get nothing except the same AI tools as anyone else. That is natural since the users are bringing the problem to the AI, of course they have the lion share here.

AI is useless until applied to a specific task owned by a person or company. Within such a task there is opportunity for AI to generate value. AI does not generate its own opportunities, users do.

Because users are distributed across society benefits follow the same curve. They don't flow to the center but mainly remain at the edge. In this sense LLMs are like Linux, they serve every user in their specific way, but the contributors to the open source code don't get directly compensated.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#254

Post by xena »

mattl wrote: post #246
xena wrote: post #243

Earlier quoted context omitted.

I make this: https://anubis.techaro.lol . I have yet to add the SQL injection or IP list layers, but I can add that to the roadmap.

The proof of work stuff feels so cryptocurrency adjacent that I've been looking at other tools for my own thing, but I've seen Anubis on other websites and it seems to do a good job.

There's a non proof of work challenge: https://anubis.techaro.lol/docs/admin/configuration/challeng...

Also: Anubis does not mine cryptocurrency. Proof of work is easy to validate on the server and economically scales poorly in the wild for abusive scrapers.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#256

Post by lurking_swe »

slenk wrote: post #241
djfivyvusn wrote: post #233

Earlier quoted context omitted.

So you're not using the parasite and that's your claim why it's not a parasite?

Dude, stop putting words in my mouth. I never said they weren't bad. Some nicer people here tried the educative approach and it worked much better. I learned about Bunny. And I keep forgetting I have a few in deSec but that has a limit. I do not understand the hostility

> I do not understand the hostility

Unfortunately I don’t think they were participating in the conversation in good faith. People can have an extreme view on _anything_…even internet / tech. They buy into a dream of 100% open source, or “open internet”, or 100% decentralized, whatever.

When this happens they may be convinced that “others” are crazy for not sharing their utopian vision. And once this point is reached, they struggle to communicate with their peers or normal people effectively. They share their strong opinions without sharing important context (how they reached those opinions), they think the topic is black and white (because they feel so strongly about the topic), or they become hostile to others that are not sharing that vision.

You are their latest victim lol. Ignore them, and carry on.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#257

Post by MichaelZuo »

Dylan16807 wrote: post #237
MichaelZuo wrote: post #214

Earlier quoted context omitted.

For a whitelist system, then by definition yes? If it’s a blacklist system, like I said I’ve not heard of any feasible solution more precise than banning huge ranges of ipv6 addresses.

> For a whitelist system, then by definition yes? A whitelist system would consider all IPv4 traffic suspicious by default too. This is not an answer to why you'd be suspicious of IPv6 in particular. > I’ve not heard of any feasible solution more precise than banning huge ranges of ipv6 addresses. Handling /56s or something like that is about the same as handling individual IPv4 addresses.

> A whitelist system would consider all IPv4 traffic suspicious by default too.

Based on what argument…?

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#258

Post by az226 »

abalashov wrote: post #101

Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defendin…

That’s the irony. Doing it now is just hampering competition and making it better for the incumbents.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#259

Post by trollbridge »

rockskon wrote: post #134
andy99 wrote: post #112

Earlier quoted context omitted.

It's cloudflare and parasites like them that will make the internet un-free. It's already happening, I'm either blocked or back to 1998 load times be cause of "checking your browser". They are destroying the internet and will make it so only people who do approved things on approved browsers (meaning let advertising companies monetize their online activity) will get real access. Cloudflare isn't solving a problem, th…

LLM scrapers have dramatically been increasing the cost of hosting various small websites. Without something being done, the data that these scrapers rely on would eventually no longer exist.

I use Cloudflare and edge caching, so it doesn’t really affect me, but the amount of LLM scraping of various static assets for apps I host is ridiculous.

We’re talking a JavaScript file of strings to respond like “login failed”, “reset your password” just over and over again. Hundreds of fetches a day, often from what appears to be the same system.

Re: Cloudflare Introduces Default Blocking of A.I. Data Scrapers

#260

Post by trollbridge »

Dylan16807 wrote: post #237
MichaelZuo wrote: post #214

Earlier quoted context omitted.

For a whitelist system, then by definition yes? If it’s a blacklist system, like I said I’ve not heard of any feasible solution more precise than banning huge ranges of ipv6 addresses.

> For a whitelist system, then by definition yes? A whitelist system would consider all IPv4 traffic suspicious by default too. This is not an answer to why you'd be suspicious of IPv6 in particular. > I’ve not heard of any feasible solution more precise than banning huge ranges of ipv6 addresses. Handling /56s or something like that is about the same as handling individual IPv4 addresses.

I try to build things to be INET6 ready, and just repeat /64s like a single host. Eventually this will probably have to broadened to /56s or /48s.
Post reply on HN