Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

371–380 of 466 posts

Re: AI companies cause most of traffic on forums

#371

Post by EVa5I7bHFq9mnYK »

throwaway290 wrote: post #293
dannyw wrote: post #279

Earlier quoted context omitted.

That’s probably a page/site rule set by the website owner. Some sites block EU IPs as the costs of complying with GDPR outweigh the gain.

I saw GDPR related blockage like literally twice in a few years and I connect from EU IP almost all the time Overload of captcha is not about GDPR... but the issue is strange. @benhurmarcel I would check if there is somebody or some company nearby abusing stuff and you got under the hammer. Maybe unscrupulous VPN company. Using a good VPN can in fact make things better (but will cost money) or if you have a place to…

I found it's best to use VPSes from young and little known hosting companies, as their IP is not yet on the blacklists.

Re: AI companies cause most of traffic on forums

#372

Post by EVa5I7bHFq9mnYK »

throwaway290 wrote: post #362
benhurmarcel wrote: post #300

Earlier quoted context omitted.

Does it work only based on the IP? I also tried from a mobile 4G connection, it’s the same.

This may be too paranoid, but if your mobile IP is persistent and phone was compromised and is serving as a proxy for bots then it could explain why your IP fell out of favor

You don't get your own external IP with the phone, it's shared, like NAT.

Re: AI companies cause most of traffic on forums

#373

Post by EVa5I7bHFq9mnYK »

Tostino wrote: post #198
echelon wrote: post #154

Earlier quoted context omitted.

Then have the users solve ARC-AGI or whatever nonsense. If the bots want your content, they'll have to solve $3,000 of compute to get it.

That only works until The benchmark questions and answers are public. Which they necessarily would be in this case.

Or maybe solve a small sha2(sha2()) leading zeroes challenge, taking ~1 second of computer time. Normal users won't notice, and bots will earn you Bitcoins :)

Re: AI companies cause most of traffic on forums

#374

Post by EVa5I7bHFq9mnYK »

buro9 wrote: post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

>> there is little to no value in giving them access to the content

If you are an online shop, for example, isn't it beneficial that ChatGPT can recommend your products? Especially given that people now often consult ChatGPT instead of searching at Google?

Re: AI companies cause most of traffic on forums

#376

Post by throwaway290 »

EVa5I7bHFq9mnYK wrote: post #372
throwaway290 wrote: post #362

Earlier quoted context omitted.

This may be too paranoid, but if your mobile IP is persistent and phone was compromised and is serving as a proxy for bots then it could explain why your IP fell out of favor

You don't get your own external IP with the phone, it's shared, like NAT.

Depends on provider/plan

Re: AI companies cause most of traffic on forums

#377

Post by AndyMcConachie »

This article claims that these big companies no longer respect robots.txt. That to me is the big problem. Back when I used to work with the Google Search Appliance it was impossible to ignore robots.txt. Since when have big known companies decided to completely ignore robots.txt?

Re: AI companies cause most of traffic on forums

#378

Post by frereubu »

ndileas wrote: post #308
smolder wrote: post #242

Earlier quoted context omitted.

We just need a browser plugin to auto-email webmasters to request access, and wait for the follow-up "access granted" email. It could be powered by AI.

Then someone will require a notarized statement of intent before you can read the recipe blog.

Now we're talking. Some kind of requirement for government-issued ID too.

Re: AI companies cause most of traffic on forums

#379

Post by Kuraj »

frereubu wrote: post #80
coldpie wrote: post #65

Earlier quoted context omitted.

You know, at this point, I wonder if an allowlist would work better.

I love (hate) the idea of a site where you need to send a personal email to the webmaster to be whitelisted.

I have not heard the word "webmaster" in such a long time

Re: AI companies cause most of traffic on forums

#380

Post by nkrisc »

alphan0n wrote: post #351
nkrisc wrote: post #344

Earlier quoted context omitted.

Lying requires intent to deceive. How have you determined their intent?

When someone says: > Oh, and of course, they don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don't give a single flying fuck about robots.txt, because why should they. Their self righteous indignation and specificity of the pretend subject of that indignation precludes any doubt about intent. This guy made a whole public statement that is verifiably f…

That may all be true. That still doesn’t mean they intentionally lied.
Post reply on HN