Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

291–300 of 466 posts

Re: AI companies cause most of traffic on forums

#291

Post by alphan0n »

denschub wrote: post #290
alphan0n wrote: post #289

Earlier quoted context omitted.

What about the CommonCrawl archives? That clearly show the same robots.txt that allows all, from September through December? You’re a phony.

Here's something for the next time you want to "expose" a phony: before linking me to your investigative source, ask for exact date-stamps when I made changes to the robots.txt and what I did, as well as when I blocked IPs. I could have told you those exactly, because all those changes are tracked in a git repo. If you asked me first, I could have answered you with the precise dates, and you would have realized that…

◔_◔

Re: AI companies cause most of traffic on forums

#292

Post by [deleted] »

x0x0 wrote: post #267
ndriscoll wrote: post #229

Earlier quoted context omitted.

Beyond something like running an ML model, what web pages are expensive (enough that 1-10 requests/second matters at all) to generate these days?

I've worked on multiple sites like this over my career. Our pages were expensive to generate, so what scraping did is blew out all our caches by yanking cold pages/images into memory. Page caches, fragment caches, image caches, but also the db working set in ram, making every single thing on the site slow.

[deleted]

Re: AI companies cause most of traffic on forums

#293

Post by throwaway290 »

dannyw wrote: post #279
benhurmarcel wrote: post #248

Earlier quoted context omitted.

Sometimes it doesn’t even give you a Captcha. I have come across some websites that block me using Cloudflare with no way of solving it. I’m not sure why, I’m in a large first-world country, I tried a stock iPhone and a stock Windows PC, no VPN or anything. That’s just no way to know.

That’s probably a page/site rule set by the website owner. Some sites block EU IPs as the costs of complying with GDPR outweigh the gain.

I saw GDPR related blockage like literally twice in a few years and I connect from EU IP almost all the time

Overload of captcha is not about GDPR...

but the issue is strange. @benhurmarcel I would check if there is somebody or some company nearby abusing stuff and you got under the hammer. Maybe unscrupulous VPN company. Using a good VPN can in fact make things better (but will cost money) or if you have a place to put your own all the better. otherwise check if you can change your IP with provider or change providers or move I guess...

not to excuse CF racket but as this thread shows the data hungry artificial stupidity leaves no choice to some sites

Re: AI companies cause most of traffic on forums

#294

Post by troupo »

ndriscoll wrote: post #276
troupo wrote: post #272

Earlier quoted context omitted.

Oh, I was a bit off. They also indexed diffs > And I mean that - they indexed every single diff on every page for every change ever made. Frequently with spikes of more than 10req/s. Of course, this made MediaWiki and my database server very unhappy, causing load spikes, and effective downtime/slowness for the human users.

Does MW not store diffs as diffs (I'd think it would for storage efficiency)? That shouldn't really require much computation. Did diffs take 30s+ to render 15-20 years ago? For what it's worth my kiwix copy of Wikipedia has a ~5ms response time for an uncached article according to Firefox. If I hit a single URL with wrk (so some caching at least with disks. Don't know what else kiwix might do) at concurrency 8, it do…

> Different code bases

Indeed ;)

> If I hit a single URL with wrk

But the bots aren't hitting a single URL

As for the diffs...

According to MediaWiki it gzips diffs [1]. So to render a previous version of the page I guess it'd have to unzip and apply all diffs in sequence to render the final version of the page.

And then it depends on how efficient the queries are at fetching etc.

[1] https://www.mediawiki.org/wiki/Manual:MediaWiki_architecture

Re: AI companies cause most of traffic on forums

#295

Post by uludag »

Terr_ wrote: post #283
m3047 wrote: post #195

Earlier quoted context omitted.

(I was going to post "run a bot motel" as a topline, but I get tired of sounding like broken record.) To generate garbage data I've had good success using Markov Chains in the past. These days I think I'd try an LLM and turning up the "heat".

Wouldn't your own LLM be overkill? Ideally one would generate decoy junk more much efficiently than these abusive/hostile attackers can steal it.

I still think this could worthwhile though for these reasons.

- One "quality" poisoned document may be able to do more damage - Many crawlers will be getting this poison, so this multiplies the effect by a lot - The cost of generation seems to be much below market value at the moment

Re: AI companies cause most of traffic on forums

#296

Post by ribadeo »

joshdavham wrote: post #265
Aeolun wrote: post #166

Earlier quoted context omitted.

That’s a problem of the serverless environment, not of not being a good netizen. Seriously, my toaster from 20 years ago could serve 1req/s

What would you recommend I do instead? Deploying a Docker container on Cloud Run sorta seemed like the logical way to deploy my micro app. Also for more context, this was the app in question (now moved to streamlit cloud): https://jreadability-demo.streamlit.app/

Skip all that jazz and write some php like it's 1998 and pay 5 bucks a month for Hostens or the equivalent... Well, that's the opposite costing side of the spectrum from serverless containerized dynamic lang runtime and a zillion paid services as a backend.

Re: AI companies cause most of traffic on forums

#297

Post by ribadeo »

kerblang wrote: post #13

It looks like various companies with resources are using available means to block AI bots - it's just that the little guys don't have that kinda stuff at their disposal. What does everybody use to avoid DDOS in general? Is it just becoming Cloudflare-or-else?

Stick tables

Re: AI companies cause most of traffic on forums

#298

Post by lofaszvanitt »

uludag wrote: post #103

I'm always curious how poisoning attacks could work. Like, suppose that you were able to get enough human users to produce poisoned content. This poisoned content would be human written and not just garbage, and would contain flawed reasoning, misjudgments, lapses of reasoning, unrealistic premises, etc. Like, I've asked ChatGPT certain questions where I know the online sources are limited and it would seem that from…

Reddit is already full of these...

Re: AI companies cause most of traffic on forums

#299

Post by benhurmarcel »

dannyw wrote: post #279
benhurmarcel wrote: post #248

Earlier quoted context omitted.

Sometimes it doesn’t even give you a Captcha. I have come across some websites that block me using Cloudflare with no way of solving it. I’m not sure why, I’m in a large first-world country, I tried a stock iPhone and a stock Windows PC, no VPN or anything. That’s just no way to know.

That’s probably a page/site rule set by the website owner. Some sites block EU IPs as the costs of complying with GDPR outweigh the gain.

One of the affected websites is a local cafe in the EU. It doesn’t make any sense to block EU IPs.

Re: AI companies cause most of traffic on forums

#300

Post by benhurmarcel »

throwaway290 wrote: post #293
dannyw wrote: post #279

Earlier quoted context omitted.

That’s probably a page/site rule set by the website owner. Some sites block EU IPs as the costs of complying with GDPR outweigh the gain.

I saw GDPR related blockage like literally twice in a few years and I connect from EU IP almost all the time Overload of captcha is not about GDPR... but the issue is strange. @benhurmarcel I would check if there is somebody or some company nearby abusing stuff and you got under the hammer. Maybe unscrupulous VPN company. Using a good VPN can in fact make things better (but will cost money) or if you have a place to…

Does it work only based on the IP?

I also tried from a mobile 4G connection, it’s the same.

Post reply on HN