Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

381–390 of 466 posts

Re: AI companies cause most of traffic on forums

#381
post #349
post #344

Earlier quoted context omitted.

Lying requires intent to deceive. How have you determined their intent?

> Lying requires intent to deceive Since when do we ask people to guess other people's intent when they have better things to show, which is called evidence? Surely we should talk about things with substantiated matter?

Because there’s a meaningful difference between being wrong and lying.

There’s evidence the statement was false, no evidence it was a lie.

Re: AI companies cause most of traffic on forums

#382

Earlier quoted context omitted.

This may be too paranoid, but if your mobile IP is persistent and phone was compromised and is serving as a proxy for bots then it could explain why your IP fell out of favor

You don't get your own external IP with the phone, it's shared, like NAT.

I get a different IPv4 and IPv6 address every time I toggle airplane mode on and off.

Re: AI companies cause most of traffic on forums

#383
post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

>> there is little to no value in giving them access to the content If you are an online shop, for example, isn't it beneficial that ChatGPT can recommend your products? Especially given that people now often consult ChatGPT instead of searching at Google?

> If you are an online shop, for example, isn't it beneficial that ChatGPT can recommend your products?

ChatGPT won't 'recommend' anything that wasn't already recommended in a Reddit post, or on an Amazon page with 5000 reviews.

You have however correctly spotted the market opportunity. Future versions of CGPT with offer the ability to "promote" your eshop in responses, in exchange for money.

Re: AI companies cause most of traffic on forums

#384
post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

Would you consider giving these crawlers access if they paid you?

No, because the price they'd offer would be insultingly low. The only way to get a good price is to take them to court for prior IP theft (as NYT and others have done), and get lawyers involved to work out a licensing deal.

Re: AI companies cause most of traffic on forums

#385

If they're not respecting robots.txt, and they're causing degradation in service, it's unauthorised access, and therefore arguably criminal behaviour in multiple jurisdictions. Honestly, call your local cyber-interested law enforcement. NCSC in UK, maybe FBI in US? Genuinely, they'll not like this. It's bad enough that we have DDoS from actual bad actors going on, we don't need this as well.

Every one of these companies is sparing no expense to tilt the justice system in their favour. "Get a lawyer" is often said here, but it's advice that's most easily doable by those that have them on retainer, as well as an army of lobbyists on Capitol Hill working to make exceptions for precisely this kind of unauthorized access .

Re: AI companies cause most of traffic on forums

#386
> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

Sounds like grounds for a criminal complaint under the CFAA.

Re: AI companies cause most of traffic on forums

#387

Earlier quoted context omitted.

> Cloudflare also has a feature to block known AI bots and even suspected AI bots In addition to other crushing internet risks, add wrongly blacklisted as a bot to the list.

This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service. [0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.

The difference between politics and diplomacy is that you can survive in politics without resorting to collective punishment.

Re: AI companies cause most of traffic on forums

#388
post #379

Earlier quoted context omitted.

I love (hate) the idea of a site where you need to send a personal email to the webmaster to be whitelisted.

I have not heard the word "webmaster" in such a long time

Deliberately chosen for the nostalgia value :)

Re: AI companies cause most of traffic on forums

#389
post #194

Earlier quoted context omitted.

I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?

I would take anything the author said with a grain of salt. They straight up lied about the configuration of the robots.txt file. https://news.ycombinator.com/item?id=42551628

How do you know what the contextual configuration of their robots.txt is/was?

Your accusation was directly addressed by the author in a comment on the original post, IIRC

i find your attitude as expressed here to be problematic in many ways

Re: AI companies cause most of traffic on forums

#390
post #34

Some of these ai companies are so aggressive they are essentially dos’ing sites offline with their request volumes. Should be careful before they get blacked and can’t get data anymore. ;)

>before they get blacked ...Please don't phrase it like that.

Its probably 'blocked' misspelled, given the context.

Not everyone speaks English as a first language

Post reply on HN