Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

361–370 of 466 posts

Re: AI companies cause most of traffic on forums

#361

Post by tecoholic »

lolinder wrote: post #314
tecoholic wrote: post #310

For any self-hosting enthusiasts out here. Check your network traffic if you have a Gitea instance running. My network traffic was mostly just AmazonBot and some others from China hitting every possible URL constantly. My traffic has gone from 2-5GB per day to a tenth of that after blocking the bots.

This is one of many reasons why I don't host on the open internet. All my stuff is running on my local network, accessible via VPN if needed.

It’s nuts. Went to bed one day and couldn’t sleep because of the fan noise coming from the cupboard. So decided to investigate the next day and stumbled into this. Madness, the kind of traffic these bots are generating and the energy waste.

Re: AI companies cause most of traffic on forums

#362

Post by throwaway290 »

benhurmarcel wrote: post #300
throwaway290 wrote: post #293

Earlier quoted context omitted.

I saw GDPR related blockage like literally twice in a few years and I connect from EU IP almost all the time Overload of captcha is not about GDPR... but the issue is strange. @benhurmarcel I would check if there is somebody or some company nearby abusing stuff and you got under the hammer. Maybe unscrupulous VPN company. Using a good VPN can in fact make things better (but will cost money) or if you have a place to…

Does it work only based on the IP? I also tried from a mobile 4G connection, it’s the same.

This may be too paranoid, but if your mobile IP is persistent and phone was compromised and is serving as a proxy for bots then it could explain why your IP fell out of favor

Re: AI companies cause most of traffic on forums

#363

Post by aorth »

Beijinger wrote: post #333
kmoser wrote: post #286

Earlier quoted context omitted.

My cheap and dirty way of dealing with bots like that is to block any IP address that accesses any URLs in robots.txt. It's not a perfect strategy but it gives me pretty good results given the simplicity to implement.

How can I implement this?

Another related idea: use fail2ban to monitor the server access logs. There is one filter that will ban hosts that request non-existent URLs like WordPress login and other PHP files. If your server is not hosting PHP at all it's an obvious sign that the requests are from bots that are probing maliciously.

Re: AI companies cause most of traffic on forums

#364

Post by aorth »

walterbell wrote: post #10

OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…

Note that the official docs from OpenAI listing their user agents and IP ranges is here: https://platform.openai.com/docs/bots

Re: AI companies cause most of traffic on forums

#365

Post by grishka »

kmeisthax wrote: post #322
MetaWhirledPeas wrote: post #114

Earlier quoted context omitted.

> Cloudflare also has a feature to block known AI bots and even suspected AI bots In addition to other crushing internet risks, add wrongly blacklisted as a bot to the list.

This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service. [0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.

I have a growing Mastodon thread of this shit: https://mastodon.social/@grishka/111934602844613193

It's of course trivially bypassable with a VPN, but getting a 403 for an innocent get request of a public resource makes me angry every time nonetheless.

Re: AI companies cause most of traffic on forums

#366

Post by alphan0n »

alphan0n wrote: post #358
denschub wrote: post #290

Earlier quoted context omitted.

Here's something for the next time you want to "expose" a phony: before linking me to your investigative source, ask for exact date-stamps when I made changes to the robots.txt and what I did, as well as when I blocked IPs. I could have told you those exactly, because all those changes are tracked in a git repo. If you asked me first, I could have answered you with the precise dates, and you would have realized that…

Gentleman’s bet. If you can accurately predict the day of four of the next six months of commoncrawls crawl, I’ll donate $500 to the charity of your choice. Fail to, donate $100 to the charity of my choice.

Or heck, $1000 to the charity of your choice if you can do 6 of 6, no expectation on your end. Just name the day from February to July, since you’re no idiot.

Re: AI companies cause most of traffic on forums

#367

Post by buro9 »

Libcat99 wrote: post #359
gs17 wrote: post #173

Earlier quoted context omitted.

From the article: > If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). It would be interesting if you had any data about this, since you seem like you would notice who behaves "better" and who tries every trick to get around blocks.

Switching to sending wrong, inexpensive data might be preferable to blocking them. I've used this with voip scanners.

Oh I did this with the Facebook one and redirected them to a 100MB file of garbage that is part of the Cloudflare speed test... they hit this so many times that it would've been 2PB sent in a matter of hours.

I contacted the network team at Cloudflare to apologise and also to confirm whether Facebook did actually follow the redirect... it's hard for Cloudflare to see 2PB, that kind of number is too small on a global scale when it's occurred over a few hours, but given that it was only a single PoP that would've handled it, then it would've been visible.

It was not visible, which means we can conclude that Facebook were not following redirects, or if they were, they were just queuing it for later and would only hit it once and not multiple times.

Re: AI companies cause most of traffic on forums

#368

Post by eesmith »

vasco wrote: post #325
eesmith wrote: post #187

Earlier quoted context omitted.

Quoting the top-level link to geraspora.de: > Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. And the best thing of all: they crawl the stupidest pages possible. Recently, both ChatGPT and Amazon were - at the same time - crawling the entire edit history…

Edit history of a wiki sounds much more interesting than the current snapshot if you want to train a model.

Does that information improve or worsen the training?

Does it justify the resource demands?

Who pays for those resources and who benefits?

Re: AI companies cause most of traffic on forums

#369

Post by CalRobert »

MetaWhirledPeas wrote: post #114
markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

> Cloudflare also has a feature to block known AI bots and even suspected AI bots In addition to other crushing internet risks, add wrongly blacklisted as a bot to the list.

We’re rapidly approaching a login-only internet. If you’re not logged in with google on chrome then no website for you!

Attestation/wei enable this

Re: AI companies cause most of traffic on forums

#370

Post by EVa5I7bHFq9mnYK »

markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

Yeah, super convenient, now every second web site blocks me as "suspected AI bot".
Post reply on HN