Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The bot situation on the internet is worse than you could imagine

gladeart.com

Post reply on HN
Print view View original

101–110 of 179 posts

Re: The bot situation on the internet is worse than you could imagine

#101

Post by PowerElectronix »

uberman wrote: post #94
siva7 wrote: post #57

So the elephant in the room: How much of HN is bot generated? Those who know have every incentive not to share and those who don't have no way to figure it out. At this point i have to assume that every new account is a bot

I've thought about this a bit and I can't really see why someone would want to write AI content here other than to spam ads but they are handled quickly. Does anyone see AI content with a clear motivation or agenda here? There are very few rep based privileges right so that seems like an unlikely motivation as well.

You can automate shilling to drive or at least influence opinion.

Re: The bot situation on the internet is worse than you could imagine

#102

Post by Aurornis »

GeoAtreides wrote: post #85
drum55 wrote: post #38

Earlier quoted context omitted.

Ironically I used a LLM to write a bypass for this ridiculous tool, doing hashing in a browser makes no sense, Claude's very bad implementation of it in C does tens of megahash a second and passes all of the challenges nearly instantly. It took about 5 minutes for Claude to write that, and it's not even a particularly fast implementation, but it beats the pants off doing string comparisons for every loop in JavaScrip…

>It's less proof of work and just annoying to users, and feel good to whoever added it to their site, this is being disproved in the article posted: >And so Anubis was enabled in the tar pit at difficulty 1 (lowest setting) when requests were pouring in 24/7. Before it was enabled, it was getting several hundred-thousand requests each day. As soon as Anubis became active in there, it decreased to about 11 requests af…

The Anubis difficulty setting is (or was) so high that nobody could visit the site without leaving it open for minutes or hours.

Re: The bot situation on the internet is worse than you could imagine

#103

Post by GeoAtreides »

Aurornis wrote: post #102
GeoAtreides wrote: post #85

Earlier quoted context omitted.

>It's less proof of work and just annoying to users, and feel good to whoever added it to their site, this is being disproved in the article posted: >And so Anubis was enabled in the tar pit at difficulty 1 (lowest setting) when requests were pouring in 24/7. Before it was enabled, it was getting several hundred-thousand requests each day. As soon as Anubis became active in there, it decreased to about 11 requests af…

The Anubis difficulty setting is (or was) so high that nobody could visit the site without leaving it open for minutes or hours.

>Anubis was enabled in the tar pit at difficulty 1 (lowest setting) when requests were pouring in 24/7

>difficulty 1 (lowest setting)

literally in the comment you're responding to

Re: The bot situation on the internet is worse than you could imagine

#104

Post by oasisbob »

simonw wrote: post #74

> These bots are almost certainly scraping data for AI training; normal bad actors don't have funding for millions of unique IPs thrown at a page. They probably belong to several different companies. Perhaps they sell their scraped data to AI companies, or they are AI companies themselves. We can't tell, but we can guess since there aren't all that many large AI corporations out there. Is the theory here that OpenAI,…

I want more data too.

The root sources of the traffic from residential proxies gets murky very quickly.

It's easy to follow the chain partway for some traffic, eg "Why are we receiving all this traffic from Digital Ocean? ... oh, it's their hero client Firecrawl, using a deceptive UserAgent" ... but it still leaves the obvious question about who the Firecrawl client is.

Res proxy traffic is insane these days. There is also plenty of grey-market snowshoe IPs available for the right price, from a handful of ASNs. I regularly see unified crawling missions by unknown agents using 1000+ "clean" IP addresses an hour.

Re: The bot situation on the internet is worse than you could imagine

#105

Post by snapetom »

siva7 wrote: post #57

So the elephant in the room: How much of HN is bot generated? Those who know have every incentive not to share and those who don't have no way to figure it out. At this point i have to assume that every new account is a bot

I think HN is one of the better ones these days. I have no data to back this up, but the comments aren't like reddit comments. Go into any reddit post on the main subs, and you won't have to scroll very far to get a comment about Trump derailing the whole thing.

Digg's recent shutdown message talked about how bad and aggressive bots were. I'd love to see Kevin and Alex post in depth about lessons learned, Dead Internet, and call out social sites.

Re: The bot situation on the internet is worse than you could imagine

#106

Post by lm411 »

AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW.

I was hit with a pretty substantial botnet "distributed scraping" attack yesterday.

- About 400,000 different IP addresses over about 3 hours

- Mostly residential IP addresses

- Valid and unique user agents and referrers

- Each IP address would make only a few requests with a long delay in between requests

It would hit the server hard until the server became slow to respond, then it would back off for about 30 seconds, then hit hard again. I was able to block most of the requests with a combination of user agent and referrer patterns, though some legit users may be blocked.

The attack was annoying, but, the even bigger problem is that the data on this website is under license - we have to pay for it, and it's not cheap. We are able to pay for it (barely) with advertising revenue and some subscriptions.

If everyone is getting this data from their "agent" and scrapers, that means no advertising revenue, and soon enough no more website to scrape, jobs lost, nowhere for scrapers to scrape for the data, nowhere for legit users to get the data for free, etc.

Re: The bot situation on the internet is worse than you could imagine

#107

Post by dspillett »

oasisbob wrote: post #63

Knew it was getting bad, but Meta's facebookexternalhit bot changed their behavior recently. In addition to pulling responses with huge amplification (40x, at least, for posting a single Facebook post to an empty audience), it's sending us traffic with fbclids in the mix. No idea why. They're also sending tons of masked traffic from their ASN (and EC2), with a fully deceptive UserAgent. The weirdest part though is th…

> it's sending us traffic with fbclids in the mix. No idea why.

The click IDs are likely to make the traffic look more like a human who has clicked a link rather than a bot? That way it gets past simple filters that explicitly let such requests in before bothering to check that the source address of the request seems to be a DC rather than a residential IP.

> citing the competitive environment

All the companies are competing to be the biggest inconvenience to everyone else while scraping as much stuff as they can.

Re: The bot situation on the internet is worse than you could imagine

#109

Post by oasisbob »

dspillett wrote: post #107
oasisbob wrote: post #63

Knew it was getting bad, but Meta's facebookexternalhit bot changed their behavior recently. In addition to pulling responses with huge amplification (40x, at least, for posting a single Facebook post to an empty audience), it's sending us traffic with fbclids in the mix. No idea why. They're also sending tons of masked traffic from their ASN (and EC2), with a fully deceptive UserAgent. The weirdest part though is th…

> it's sending us traffic with fbclids in the mix. No idea why. The click IDs are likely to make the traffic look more like a human who has clicked a link rather than a bot? That way it gets past simple filters that explicitly let such requests in before bothering to check that the source address of the request seems to be a DC rather than a residential IP. > citing the competitive environment All the companies are c…

> The click IDs are likely to make the traffic look more like a human who has clicked a link rather than a bot?

It's certainly possible. However, the traffic is still coming from Facebook's network with a FB proxy PTR record in DNS. Seems much more likely to fool your typical site owner than a bad actor.

Re: The bot situation on the internet is worse than you could imagine

#110

Post by Retr0id »

uberman wrote: post #94
siva7 wrote: post #57

So the elephant in the room: How much of HN is bot generated? Those who know have every incentive not to share and those who don't have no way to figure it out. At this point i have to assume that every new account is a bot

I've thought about this a bit and I can't really see why someone would want to write AI content here other than to spam ads but they are handled quickly. Does anyone see AI content with a clear motivation or agenda here? There are very few rep based privileges right so that seems like an unlikely motivation as well.

Most of the HN bot accounts I see have a link-to-vibecoded-product in bio, and/or are trying to build up "organic" activity before a Show HN post for the same.

A less publicly-visible motive would be if they were building up accounts to use for paid-upvote schemes.

Post reply on HN