Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

401–410 of 466 posts

Re: AI companies cause most of traffic on forums

#401

Post by alphan0n »

nkrisc wrote: post #380
alphan0n wrote: post #351

Earlier quoted context omitted.

When someone says: > Oh, and of course, they don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don't give a single flying fuck about robots.txt, because why should they. Their self righteous indignation and specificity of the pretend subject of that indignation precludes any doubt about intent. This guy made a whole public statement that is verifiably f…

That may all be true. That still doesn’t mean they intentionally lied.

What is the criteria of an intentional lie, then? Admission?

The author responded:

>denschub 2 days ago [–]

>the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt

Which is verifiably untrue:

HTTP/1.1 200 server: nginx/1.27.2 date: Tue, 10 Dec 2024 13:37:20 GMT content-type: text/plain last-modified: Fri, 13 Sep 2024 18:52:00 GMT etag: W/"1c-62204b7e88e25" alt-svc: h3=":443", h2=":443" X-Crawler-content-encoding: gzip Content-Length: 28

User-agent: * Disallow: /w/

Re: AI companies cause most of traffic on forums

#402

Post by citrin_ru »

eckesicle wrote: post #328
ls612 wrote: post #324

Earlier quoted context omitted.

People hate collective punishment because it works so well.

Anecdatally, by default, we now block all Chinese and Russian IPs across our servers. After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.

Being slightly annoyed by noise in SSH logs I’ve blocked APNIC IPs and now see a comparable number of brute force attempts from ARIN IPs (mostly US ones). Geo blocks are totally ineffective against TAs which use a global network of proxies.

Re: AI companies cause most of traffic on forums

#403

Post by Nevermark »

throwaway_fai wrote: post #142
BehindBlueEyes wrote: post #138

Earlier quoted context omitted.

Can you even DMCS takedown crawlers?

Doubt it, a vanilla cease-and-desist letter would probably be the approach there. I doubt any large AI company would pay attention though, since, even if they're in the wrong, they can outspend almost anyone in court.

Small claims court?

Re: AI companies cause most of traffic on forums

#404

Post by trod1234 »

IncreasePosts wrote: post #313
Capricorn2481 wrote: post #305

Earlier quoted context omitted.

I don't understand this. You don't have routes your users might need in robots.txt? This article is about bots accessing resources that other might use.

It seems better to put fake honeypot urls in robots.txt, and block any up that accesses those.

Blocking will never work.

You need to impose cost. Set up QoS buckets, slow suspect connections down dramatically (almost to the point of timeout).

Re: AI companies cause most of traffic on forums

#405

Post by [deleted] »

benhurmarcel wrote: post #248
throwaway290 wrote: post #119

Earlier quoted context omitted.

What do you mean crushing risk? Just solve these 12 puzzles by moving tiny icons on tiny canvas while on the phone and you are in the clear for a couple more hours!

Sometimes it doesn’t even give you a Captcha. I have come across some websites that block me using Cloudflare with no way of solving it. I’m not sure why, I’m in a large first-world country, I tried a stock iPhone and a stock Windows PC, no VPN or anything. That’s just no way to know.

[deleted]

Re: AI companies cause most of traffic on forums

#406

Post by brianwawok »

shark_laser wrote: post #348
kmeisthax wrote: post #322

Earlier quoted context omitted.

This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service. [0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.

Yep. Same for most of Asia too. Cloudflare's filters are basically straight up racist. I have stopped using so many sites due to their use of Cloudflare.

If 90% of your problem users come from 1-2 countries, seems pretty sensible to block that country. I know I have 0 paying users in those countries, so why deal with it? Let them go fight it out doing bot wars in local sites

Re: AI companies cause most of traffic on forums

#407

Post by brianwawok »

dgfitz wrote: post #331
eckesicle wrote: post #328

Earlier quoted context omitted.

Anecdatally, by default, we now block all Chinese and Russian IPs across our servers. After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.

[flagged]

That is not at all the reason for the great firewall.

Re: AI companies cause most of traffic on forums

#408

Post by xp84 »

TacticalCoder wrote: post #392
eckesicle wrote: post #328

Earlier quoted context omitted.

Anecdatally, by default, we now block all Chinese and Russian IPs across our servers. After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.

> Anecdatally, by default, we now block all Chinese and Russian IPs across our servers. This. Just get several countries' entire IP address space and block these. I've posted I was doing just that only to be told that this wasn't in the "spirit" of the Internet or whatever similar nonsense. In addition to that only allow SSH in from the few countries / ISPs legit trafic shall legitimately be coming from. This quiets…

I agree with your approach. It’s easy to empathize with innocent people in say, Russia, blocked from a site which has useful information to them. However the thing these “spirit/openness” people miss is that many sites have a narrow purpose which makes no sense to open it up to people across the world. For instance, local government. Nobody in India or Russia needs to see the minutes from some US city council meeting, or get building permit information. Likewise with e-commerce. If I sell chocolate bars and ship to US and Canada, why wouldn’t I turn off all access from overseas? You might say “oh, but what if some friend in $COUNTRY wants to order a treat for someone here?” And the response to that is always “the hypothetical loss from that is minuscule compared to the cost of serving tons of bot traffic as well as possible exploits those bots might do.

(Yes, yes, VPNs and proxies exist and can be used by both good and bad actors to evade this strategy, and those are another set of IPs widely banned for the same reason. It’s a cat and mouse game but you can’t argue with the results)

Re: AI companies cause most of traffic on forums

#409

Post by crote »

d0mine wrote: post #391
kmeisthax wrote: post #322

Earlier quoted context omitted.

This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service. [0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.

unrelated: USSR might have been 2nd world. Russia is 3rd world (since 1991) -- banana republic

No, Russia is by definition the 2nd world. It's about spheres of influence, not any kind of economic status. The First World is the Western Bloc centered around the US, the Second World is the Eastern Bloc centered around then-USSR and now-Russia (although these days more centered on China), the Third World is everyone else.

Re: AI companies cause most of traffic on forums

#410

Post by arcastroe »

> If you try to block them by User Agent string, they'll just switch to a non-bot UA string (no, really).

Instead of blocking them (non-200 response), what if you shadow-ban them and instead serve 200-response with some useless static content specifically made for the bots?

Post reply on HN