Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The bot situation on the internet is worse than you could imagine

gladeart.com

Post reply on HN
Print view View original

111–120 of 179 posts

Re: The bot situation on the internet is worse than you could imagine

#111

Post by wiseowise »

lm411 wrote: post #106

AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW. I was hit with a pretty substantial botnet "distributed scraping" attack yesterday. - About 400,000 different IP addresses over about 3 hours - Mostly residential IP addresses - Valid and unique user agents and referrers - Each IP address would make only a few requests with a long delay in between requests It would hit the se…

Don’t worry, man, once AGI is here you’ll get your allowance (or whatever the hyperscalers plan is).

Re: The bot situation on the internet is worse than you could imagine

#112

Post by everdrive »

lm411 wrote: post #106

AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW. I was hit with a pretty substantial botnet "distributed scraping" attack yesterday. - About 400,000 different IP addresses over about 3 hours - Mostly residential IP addresses - Valid and unique user agents and referrers - Each IP address would make only a few requests with a long delay in between requests It would hit the se…

Thanks for sharing the perspective here. I think a lot of folks on HN have rightly said that a lot of the problems with the modern internet are due to the ad-supported business model. I don't think you were ever going to move away from it voluntarily -- too many people support it, even if they grumble about it.

But maybe (and likely for worse) LLMs will finally kill this model.

Re: The bot situation on the internet is worse than you could imagine

#113

Post by marginalia_nu »

simonw wrote: post #74

> These bots are almost certainly scraping data for AI training; normal bad actors don't have funding for millions of unique IPs thrown at a page. They probably belong to several different companies. Perhaps they sell their scraped data to AI companies, or they are AI companies themselves. We can't tell, but we can guess since there aren't all that many large AI corporations out there. Is the theory here that OpenAI,…

I agree it's a more than a bit handwavy. The common consensus seems to be that AI companies are driving this, but it's really hard to conclusively prove who or what is behind the attacks.

Weird part #1 is that the traffic isn't for the most part shaped like crawler traffic. It's incredibly bursty, and heavily redundant, missing even the most obvious low hanging fruit optimizations.

Could be someone is using residential proxies to wrap AI agents' web traffic, but even so, there's a lot of pieces that don't really make sense, like why the traffic pattern is like being hit by a shotgun. It isn't just one request, but anywhere between 40 and 100 redundant requests.

A popular theory is that this is because of sloppy coding, AI companies are too rich to care, but then again that doesn't really add up. This isn't just a minor inefficiency, if it is "just" bad coding, they stand to gain monumental efficiency improvements by fixing the issues, in the sense of getting the data much faster, a clear competitive edge.

Really weird.

My unsubstantiated guess is the residential proxy/botnet is very unreliable, and that's why they fire so many request. Makes sense if it's sold as a service.

Re: The bot situation on the internet is worse than you could imagine

#114

Post by shimman »

lm411 wrote: post #106

AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW. I was hit with a pretty substantial botnet "distributed scraping" attack yesterday. - About 400,000 different IP addresses over about 3 hours - Mostly residential IP addresses - Valid and unique user agents and referrers - Each IP address would make only a few requests with a long delay in between requests It would hit the se…

Do you not run Anubis or have strict fail2ban rules? I just straight up ban IPs forever if they lookup files that will never exist on my servers. That plus Anubis with the strictest settings.

https://anubis.techaro.lol/

Re: The bot situation on the internet is worse than you could imagine

#115

Post by rekabis »

lifeisstillgood wrote: post #91
rekabis wrote: post #32

Earlier quoted context omitted.

If you want “papers, please” every time you back out of your driveway or go beyond your government-assigned oblast, then your suggestion is the digital version of the physical authoritarian nightmare that was imposed by totalitarianist regimes throughout history. People have a right to complete anonymity, and should be able to go across the majority of the Internet just as they can go across most of the country. That…

>>> People have a right to complete anonymity Why? (Am not trolling. Genuinely interested) I walk out my front door in the UK and I am not anonymous. Every transaction I make either identifies me through bank, railway or other id, or quite simply by my face standing in front of the coffee seller. My walk down the road is observed by neighbours and postmen. Should my government arrest me without cause or trample on my…

> Every transaction I make either identifies me through bank, railway or other id, or quite simply by my face standing in front of the coffee seller. My walk down the road is observed by neighbours and postmen.

Are these the government? Is the bank the government? Is the rail company the government?

No? Then you have answered your own question.

A silo of identification between you and a service provider that uses the provider’s own tooling is still anonymity from government authoritarianism.

The fact that nearly all of these silos are leaky IRL - with the government eager to punch howitzer-sized holes through them for even more access - is not the point. It is a citizen-hostile flaw that needs patching through loophole-proof legislation, not an ID system that would violently eradicate any remaining separation of government from capitalism.

Remember: when government and capitalism rides in the same cart, it is called corporatism, and is the basis of Fascism. Which is what is happening to America.

Re: The bot situation on the internet is worse than you could imagine

#116

Post by afinlayson »

lm411 wrote: post #106

AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW. I was hit with a pretty substantial botnet "distributed scraping" attack yesterday. - About 400,000 different IP addresses over about 3 hours - Mostly residential IP addresses - Valid and unique user agents and referrers - Each IP address would make only a few requests with a long delay in between requests It would hit the se…

At some point there needs to be a check if it's a real human... But it's a cat and mouse game - any way we create to keep bots off gets a work around by clever engineers.

Re: The bot situation on the internet is worse than you could imagine

#118

Post by braingravy »

gruez wrote: post #77
timshell wrote: post #55

My grad school research was on computational models of human/machine cognition, and I'm now commercializing it as a 'proof-of-human API' for bot detection, spam reduction, and identity verification. One of the mistakes people assume is that AI capability means humanness. If you know exactly where to look, you can start to identify differences between improving frontier models and human cognition. One concrete example…

>The first problem can be reduced to a machine learning classifier. The second is an unsolved scientific problem. I can't believe people are still using this as a generic anti-AI argument even though a decade ago people were insisting that there's no way AI can have the capabilities that frontier LLMs have today. Moreover it's unclear whether the gap even exists. Even if we take the claim that the grid pattern is som…

> “…reverse-engineering the computational dynamics of a biological brain and reproducing them in real time…”

Is not an anti-AI argument, it’s an open and unsolved question. Your optimism is appreciated, but the dismissal and assumption this is already solved is foolish and naive.

Re: The bot situation on the internet is worse than you could imagine

#119

Post by righthand »

wiseowise wrote: post #111
lm411 wrote: post #106

AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW. I was hit with a pretty substantial botnet "distributed scraping" attack yesterday. - About 400,000 different IP addresses over about 3 hours - Mostly residential IP addresses - Valid and unique user agents and referrers - Each IP address would make only a few requests with a long delay in between requests It would hit the se…

Don’t worry, man, once AGI is here you’ll get your allowance (or whatever the hyperscalers plan is).

You’ll enjoy painting or some other art even if you aren’t interested in the arts. That’s what I’ve seen written about it.

Re: The bot situation on the internet is worse than you could imagine

#120

Post by oasisbob »

marginalia_nu wrote: post #113
simonw wrote: post #74

> These bots are almost certainly scraping data for AI training; normal bad actors don't have funding for millions of unique IPs thrown at a page. They probably belong to several different companies. Perhaps they sell their scraped data to AI companies, or they are AI companies themselves. We can't tell, but we can guess since there aren't all that many large AI corporations out there. Is the theory here that OpenAI,…

I agree it's a more than a bit handwavy. The common consensus seems to be that AI companies are driving this, but it's really hard to conclusively prove who or what is behind the attacks. Weird part #1 is that the traffic isn't for the most part shaped like crawler traffic. It's incredibly bursty, and heavily redundant, missing even the most obvious low hanging fruit optimizations. Could be someone is using residenti…

> A popular theory is that this is because of sloppy coding, AI companies are too rich to care, but then again that doesn't really add up

I can substantiate this a bit. Verified traffic from Amazonbot is too dumb to do anything with 429s. They will happily slam your site with more traffic than you can handle, and will completely ignore the fact that over half the responses are useless rate limits.

They say they honor REP, but Amazonbot will still hit you pretty persistently even with a full disallow directive in robots.txt

Post reply on HN