Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

411–420 of 466 posts

Re: AI companies cause most of traffic on forums

#411

Post by lazide »

shark_laser wrote: post #348
kmeisthax wrote: post #322

Earlier quoted context omitted.

This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service. [0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.

Yep. Same for most of Asia too. Cloudflare's filters are basically straight up racist. I have stopped using so many sites due to their use of Cloudflare.

Well, not racist per-se - if you visit the countries (regardless of race) you’re screwed too.

Geo-location-ist?

Re: AI companies cause most of traffic on forums

#412

Post by lazide »

scarface_74 wrote: post #382
EVa5I7bHFq9mnYK wrote: post #372

Earlier quoted context omitted.

You don't get your own external IP with the phone, it's shared, like NAT.

I get a different IPv4 and IPv6 address every time I toggle airplane mode on and off.

Externally routable IPv4, or just a different between-a-cgnat address?

Re: AI companies cause most of traffic on forums

#413

Post by lazide »

llm_trw wrote: post #179
echelon wrote: post #129

Earlier quoted context omitted.

You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy. Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills. Maybe you don't even need a full LLM. Just a simple transfor…

You will be burning through thousands of dollars worth of compute to do that.

The biggest issue is at least 80% of internet users won’t be capable of passing the test.

Re: AI companies cause most of traffic on forums

#414

Post by phendrenad2 »

ksec wrote: post #87

On a side note. CGTalk ( One of the largest forum for 3D / VFX Content creation ) was shut down earlier this year citing resources reasons. And for whatever reason they didn't allow anyone to take a copy of the data, run the forum. They could have sold that Data to OpenAI. I wonder how many Forums shut down due to traffics like this? Most of the reason why forums moved to Slack, Discord etc was that they no longer ha…

I doubt OpenAI would buy the data, they probably scraped it already.

Looks like CGTalk was running VBulletin until 2018, when they switched to Discourse. Discourse is a huge step down in terms of usability and polish, but I can understand why they potentially did that. VBulletin gets expensive to upgrade, and is a big modular system like wordpress, so you have to keep it patched or you will likely get hacked.

Bottom-line is running a forum in 2024 requires serious commitment.

Re: AI companies cause most of traffic on forums

#415

Post by scarface_74 »

lazide wrote: post #412
scarface_74 wrote: post #382

Earlier quoted context omitted.

I get a different IPv4 and IPv6 address every time I toggle airplane mode on and off.

Externally routable IPv4, or just a different between-a-cgnat address?

Externally routable IPv4 as seen by whatismyip.com.

Re: AI companies cause most of traffic on forums

#416

Post by yupyupyups »

layer8 wrote: post #192
uludag wrote: post #103

I'm always curious how poisoning attacks could work. Like, suppose that you were able to get enough human users to produce poisoned content. This poisoned content would be human written and not just garbage, and would contain flawed reasoning, misjudgments, lapses of reasoning, unrealistic premises, etc. Like, I've asked ChatGPT certain questions where I know the online sources are limited and it would seem that from…

Unless a substantial portion of the internet starts serving poisoned content to bots, that won’t solve the bandwidth problem. And even if a substantial portion of the internet would start poisoning, bots would likely just shift to disguising themselves so they can’t be identified as bots anymore. Which according to the article they already do now when they are being blocked.

>even if a substantial portion of the internet would start poisoning, bots would likely just shift to disguising themselves so they can’t be identified as bots anymore.

Good questions to ask would be:

- How do they disguise themselves?

- What fundamental features do bots have that distinguish them from real users?

- Can we use poisoning in conjunction with traditional methods like a good IP block lists to remove the low hanging fruits?

Re: AI companies cause most of traffic on forums

#417

Post by ranger207 »

nedrocks wrote: post #213
ai-christianson wrote: post #122

Earlier quoted context omitted.

Would you consider giving these crawlers access if they paid you?

This is one of the few interesting uses of crypto transactions at reasonable scale in the real world.

What does crypto add here that can't be accomplished with regular payments?

Re: AI companies cause most of traffic on forums

#418

Post by tliltocatl »

buro9 wrote: post #367
Libcat99 wrote: post #359

Earlier quoted context omitted.

Switching to sending wrong, inexpensive data might be preferable to blocking them. I've used this with voip scanners.

Oh I did this with the Facebook one and redirected them to a 100MB file of garbage that is part of the Cloudflare speed test... they hit this so many times that it would've been 2PB sent in a matter of hours. I contacted the network team at Cloudflare to apologise and also to confirm whether Facebook did actually follow the redirect... it's hard for Cloudflare to see 2PB, that kind of number is too small on a global…

Hmm, what about 1kb of carefully crafted gz-bomb? Or a TCP tarpit (this one would be a bit difficult to deploy).

Re: AI companies cause most of traffic on forums

#419

Post by PeterStuer »

markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

Most administrators have no idea or no desire to correctly configure Cloudflare, so they just slap it on the whole site by default and block all the legitimate access to e.g. rss feeds.

Re: AI companies cause most of traffic on forums

#420

Post by buro9 »

l1n wrote: post #280
buro9 wrote: post #99

Earlier quoted context omitted.

Nginx, it's nothing special it's just my load balancer. if ($http_user_agent ~* (list|of|case|insensitive|things|to|block)) {return 403;}

403 is generally a bad way to get crawlers to go away - https://developers.google.com/search/blog/2023/02/dont-404-m... suggests a 500, 503, or 429 HTTP status code.

> 403 is generally a bad way to get crawlers to go away

Hardly... the article links says that a 403 will cause Google to stop crawling and remove content... that's the desired outcome.

I'm not trying to rate limit, I'm telling them to go away.

Post reply on HN