Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The bot situation on the internet is worse than you could imagine

gladeart.com

Post reply on HN
Print view View original

121–130 of 179 posts

Re: The bot situation on the internet is worse than you could imagine

#121

Post by marginalia_nu »

oasisbob wrote: post #120
marginalia_nu wrote: post #113

Earlier quoted context omitted.

I agree it's a more than a bit handwavy. The common consensus seems to be that AI companies are driving this, but it's really hard to conclusively prove who or what is behind the attacks. Weird part #1 is that the traffic isn't for the most part shaped like crawler traffic. It's incredibly bursty, and heavily redundant, missing even the most obvious low hanging fruit optimizations. Could be someone is using residenti…

> A popular theory is that this is because of sloppy coding, AI companies are too rich to care, but then again that doesn't really add up I can substantiate this a bit. Verified traffic from Amazonbot is too dumb to do anything with 429s. They will happily slam your site with more traffic than you can handle, and will completely ignore the fact that over half the responses are useless rate limits. They say they honor…

How do you know it's Amazonbot?

Re: The bot situation on the internet is worse than you could imagine

#122

Post by gamesieve »

marginalia_nu wrote: post #113
simonw wrote: post #74

> These bots are almost certainly scraping data for AI training; normal bad actors don't have funding for millions of unique IPs thrown at a page. They probably belong to several different companies. Perhaps they sell their scraped data to AI companies, or they are AI companies themselves. We can't tell, but we can guess since there aren't all that many large AI corporations out there. Is the theory here that OpenAI,…

I agree it's a more than a bit handwavy. The common consensus seems to be that AI companies are driving this, but it's really hard to conclusively prove who or what is behind the attacks. Weird part #1 is that the traffic isn't for the most part shaped like crawler traffic. It's incredibly bursty, and heavily redundant, missing even the most obvious low hanging fruit optimizations. Could be someone is using residenti…

I suspect the redundant requests are primarily designed to weed out poisoned data served on otherwise valid URLs. I've also seen the redundant requests increase massively the more sources I blocked at the firewall level, so it feels like they're pre-emptively overcompensating for some percentage of requests being blocked.

My website contains ~6000 unique data points in effectively infinite combinations on effectively infinite pages. Some of those combinations are useful for humans, but the AI-scrapers could gain a near-infinite efficiency improvement by just identifying as a bot and heeding my robots.txt and/or rel="nofollow" hints to access the ~500 top level pages which contain close to everything which is unique. They just don't care. All their efficiency attempts are directed solely toward bypassing blocks. (Today I saw them varying the numbers in their user agent strings: X15 rather than X11, Chrome/532 rather than Chrome/132, and so on...)

Re: The bot situation on the internet is worse than you could imagine

#123

Post by oasisbob »

marginalia_nu wrote: post #121
oasisbob wrote: post #120

Earlier quoted context omitted.

> A popular theory is that this is because of sloppy coding, AI companies are too rich to care, but then again that doesn't really add up I can substantiate this a bit. Verified traffic from Amazonbot is too dumb to do anything with 429s. They will happily slam your site with more traffic than you can handle, and will completely ignore the fact that over half the responses are useless rate limits. They say they honor…

How do you know it's Amazonbot?

User Agent, SWIPed IP space, and the PTR records resolving to an Amazon-controlled crawl zone.

Re: The bot situation on the internet is worse than you could imagine

#125

Post by lm411 »

shimman wrote: post #114
lm411 wrote: post #106

AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW. I was hit with a pretty substantial botnet "distributed scraping" attack yesterday. - About 400,000 different IP addresses over about 3 hours - Mostly residential IP addresses - Valid and unique user agents and referrers - Each IP address would make only a few requests with a long delay in between requests It would hit the se…

Do you not run Anubis or have strict fail2ban rules? I just straight up ban IPs forever if they lookup files that will never exist on my servers. That plus Anubis with the strictest settings. https://anubis.techaro.lol/

Fail2ban doesn't scale well to these volumes of traffic and request patterns.

Just like fail2ban is not very useful against a DDOS attack where each unique IP only makes a few requests with a large (hour+) delay in between requests. There is no clear "fail" in these requests, and the fail2ban database becomes huge and far too slow.

- 400,000 Unique IP addresses

- 1 to 3 requests per hour per IP addresses - with delays of over 60 minutes between each request.

- Legit request URLs, legit UA & referrer

Maybe Anubis would help, but it's also a risk for various reasons.

Re: The bot situation on the internet is worse than you could imagine

#126

Post by lifeisstillgood »

Supermancho wrote: post #99
lifeisstillgood wrote: post #91

Earlier quoted context omitted.

>>> People have a right to complete anonymity Why? (Am not trolling. Genuinely interested) I walk out my front door in the UK and I am not anonymous. Every transaction I make either identifies me through bank, railway or other id, or quite simply by my face standing in front of the coffee seller. My walk down the road is observed by neighbours and postmen. Should my government arrest me without cause or trample on my…

> Should my government arrest me without cause or trample on my free speech rights, I get that’s a problem but I am not sure why being anonymous helps. Having rights upheld by the courts helps, well trained police who respect the law helps. You're suggesting the same government that would violate your rights would then help prevent it? I don't follow. Any power structure (tiered or not) was wiped away by authoritaria…

Sorry maybe I should be clearer - the problem of tyrannical governments is not solved by being anonymous online, or indeed any technology that makes it hard for government to do the tyrannical things. Safety lies in an engaged citizenry that reacts to fundamental threats. The protests against the ICE in Minnesota being an example.

>>> It will result in all sorts of changes to cultural norms across western nations

I quite agree - but I (hope / think) that the benefits can outweighs the downsides if done well. Those nations that do it well will I believe find a rocket like boost to society and industry perhaps akin to post 1945 world. Those who don’t will fall behind.

Re: The bot situation on the internet is worse than you could imagine

#127

Post by johneth »

jwr wrote: post #61

An interesting and sad aspect of the war on bots and scraping that is being waged is that we are hurting ourselves in the process, too. Many tasks I'm trying to get my AI assistant to do cannot be done quickly, because sites defensively prohibit access to their content. I'm not scraping: it's my agent trying to fetch a page or two to perform a task for me (such as check pricing or availability). We need a better solu…

I would assume most sites that block access to your AI assistant do so because they want to show a human ads, i.e. not run at a loss. Seems reasonable.

Re: The bot situation on the internet is worse than you could imagine

#128

Post by bawolff »

GeoAtreides wrote: post #85
drum55 wrote: post #38

Earlier quoted context omitted.

Ironically I used a LLM to write a bypass for this ridiculous tool, doing hashing in a browser makes no sense, Claude's very bad implementation of it in C does tens of megahash a second and passes all of the challenges nearly instantly. It took about 5 minutes for Claude to write that, and it's not even a particularly fast implementation, but it beats the pants off doing string comparisons for every loop in JavaScrip…

>It's less proof of work and just annoying to users, and feel good to whoever added it to their site, this is being disproved in the article posted: >And so Anubis was enabled in the tar pit at difficulty 1 (lowest setting) when requests were pouring in 24/7. Before it was enabled, it was getting several hundred-thousand requests each day. As soon as Anubis became active in there, it decreased to about 11 requests af…

That doesnt mean the PoW is doing anything, it might just mean bots have js disabled.

I dont think the person was claiming annubis doesnt work, they were disputing PoW is the reason it actually works.

Re: The bot situation on the internet is worse than you could imagine

#129

Post by lm411 »

everdrive wrote: post #112
lm411 wrote: post #106

AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW. I was hit with a pretty substantial botnet "distributed scraping" attack yesterday. - About 400,000 different IP addresses over about 3 hours - Mostly residential IP addresses - Valid and unique user agents and referrers - Each IP address would make only a few requests with a long delay in between requests It would hit the se…

Thanks for sharing the perspective here. I think a lot of folks on HN have rightly said that a lot of the problems with the modern internet are due to the ad-supported business model. I don't think you were ever going to move away from it voluntarily -- too many people support it, even if they grumble about it. But maybe (and likely for worse) LLMs will finally kill this model.

I would love for the ad-supported model to die. I hate ads, and I hate having to serve ads. We get some subscription users but nowhere near enough to cover costs.

Unfortunately, what I think will happen - and indeed already is - is that the AI companies themselves will replace much of the WWW. Sites like the one I am talking about will cease to exist. AI companies, once they can no longer scrape (steal) the data will end up licensing the data themselves and replace us as the distributor to end users. Perhaps as a subscription add-on or also with an ad based model.

Which to some may be fine. Personally, I don't want a few centralized AI companies replacing the hundreds of thousands of independent websites online. Way too much centralized power there.

Re: The bot situation on the internet is worse than you could imagine

#130

Post by bawolff »

drum55 wrote: post #59
bawolff wrote: post #58

Earlier quoted context omitted.

Shouldnt browser also have it implemented in c? Like i assume crypto.subtle isnt written in js.

It doesn't matter if your hottest loop is using string comparisons, as another poster pointed out in C you aren't even doing the majority of the second hash because you know the result (or enough of it) before finishing it. The JavaScript version just does whole hashes and turns them into a Uint8Array, then iterates through it.

I dont understand what you mean. Are you saying that in C you only calculate the first few digits of the hash? That's not how sha256 works.

Edit: oh i think you mean in c the string comparison short curcuits. I would expect the same to be true in javascript too. Its true in most languages.

Maybe you are just worried about general language overhead, which is a fair point. Is the anubis check even using multiple threads? For the c case, the real benefit wouldn't be if you can use C, but if you can use the GPU.

The whole thing is kind of silly though. SHA256 is a terrible choice of hash for PoW. They should be using argon2 or something memory heavy.

Post reply on HN