Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feed the bots

maurycyz.com

Post reply on HN
Print view View original

201–210 of 216 posts

Re: Feed the bots

#201

Post by moebrowne »

maurycyz wrote: post #134
mcdeltat wrote: post #123

Maybe a dumb question but what exactly is wrong with banning the IPs? Even if the bots get more IPs over time, surely storing a list of bans is cheaper than serving content? Is the worry that the bots will eventually cycle through so many IP ranges that you end up blocking legit users?

It's often one IP (v4!) per one request. It's insane how many resources are being burned on this stupidity. Part of the reason I did this is to get good numbers on how bad the problem is: A link maze is a great way to make otherwise very stealthy bots expose themselves.

Even if this is true how long can that be sustained before they start to be recycled? I bet the scrappers make a whole lot more requests than they have IPs

Re: Feed the bots

#202

Post by yupyupyups »

I love it. Keep feeding them that slop.

A thought though. What happens if one of the bot operators sees the random stuff?

Do you think they will try to bypass it and put you and them in a cat and mouse game? Or would that be too time-consuming and unlikely?

Re: Feed the bots

#203

Post by moebrowne »

csomar wrote: post #157
mcdeltat wrote: post #123

Maybe a dumb question but what exactly is wrong with banning the IPs? Even if the bots get more IPs over time, surely storing a list of bans is cheaper than serving content? Is the worry that the bots will eventually cycle through so many IP ranges that you end up blocking legit users?

They are usually using residential IPs through SOCK5. I am not sure how they are getting these residential IPs but it is definitively suspicious. So by blocking these IPs, you are blocking your users. (ie: in many coffeshops, I get the "IP Blocked" banner, my guess is that they are running software on unsuspecting users to route this traffic).

> So by blocking these IPs, you are blocking your users.

There were 122 million residential internet connections in the US in 2024 so for an app with 1 million users the chance of affecting a single user is [1] https://docs.fcc.gov/public/attachments/DOC-411463A1.pdf

Re: Feed the bots

#204

Post by moebrowne »

kalkin wrote: post #126

I don't think this robots.txt is valid: User-agent: Googlebot PetalBot Bingbot YandexBot Kagibot Disallow: /bomb/\* Disallow: /bomb Disallow: /babble/\* Sitemap: https://maurycyz.com/sitemap.xml I think this is telling the bot named "Googlebot PetalBot Bingbot YandexBot Kagibot" - which doesn't exist - to not visit those URLs. All other bots are allowed to visit those URLs. User-Agent is supposed to be one per line,…

You're correct, it should read

    User-agent: Googlebot
    User-agent: PetalBot
    User-agent: Bingbot
    User-agent: YandexBot
    User-agent: Kagibot
    Disallow: /bomb/*
    Disallow: /bomb
    Disallow: /babble/*
    
    Sitemap: https://maurycyz.com/sitemap.xml

Re: Feed the bots

#205

Post by dilyevsky »

63stack wrote: post #167
dilyevsky wrote: post #156

Earlier quoted context omitted.

LLMs already train on mostly garbage - you are just wasting your time. Same as talking to spam callers.

There are multiple people claiming this in this thread, but with no more than a "it doesn't work stop". Would be great to hear some concrete information.

I am not actually claiming that it’s easy to filter out like the others. What Im saying is you can literally feed a ton of garbage into a training run and amazingly it still learns

Re: Feed the bots

#206

Post by 63stack »

michaelcampbell wrote: post #182
63stack wrote: post #170

Earlier quoted context omitted.

There are multiple people claiming this in this thread, but with no more than a "it doesn't work stop". Would be great to hear some concrete information.

Was saying this 3x in this thread necessary?

I'm just interested in opinions from all 3

Re: Feed the bots

#207

Post by xphos »

flir wrote: post #195
vintermann wrote: post #187

Earlier quoted context omitted.

So this particular LLM wasn't fooled. But what happens if you ask it to summarize the Markov generated pages without warning it what it is?

They don't seem to contain the meta-instruction. Example with GPT-5 Thinking: https://maurycyz.com/babble/With-Wand-and-Broomstick-Across-... "Short version: The page is a nonsense / glitchy faux-technical text dressed up like a lab manual." If we take a page from the site that doesn't hint but does have the in-band instruction: https://maurycyz.com/misc/ipv4/ "Short version: I tried to load https://maurycyz.com/misc…

I mean how does it know that though? How would you know if the set of possible texts is garbage without running them? Honestly feels like your saying LLMs solved the halting problem as programs which seems to be dishonest granted you could probably guess with high efficiency

Re: Feed the bots

#208

Post by flir »

xphos wrote: post #207
flir wrote: post #195

Earlier quoted context omitted.

They don't seem to contain the meta-instruction. Example with GPT-5 Thinking: https://maurycyz.com/babble/With-Wand-and-Broomstick-Across-... "Short version: The page is a nonsense / glitchy faux-technical text dressed up like a lab manual." If we take a page from the site that doesn't hint but does have the in-band instruction: https://maurycyz.com/misc/ipv4/ "Short version: I tried to load https://maurycyz.com/misc…

I mean how does it know that though? How would you know if the set of possible texts is garbage without running them? Honestly feels like your saying LLMs solved the halting problem as programs which seems to be dishonest granted you could probably guess with high efficiency

> I mean how does it know that though?

Not a clue. But apparently it does. Try a few nonsense texts yourself, see if it rejects them.

I'm saying that if you're spidering the whole web, then training an LLM on that corpus, asking an existing LLM "does this page make sense?" is a comparatively small additional load.

> guess with high efficiency

Yes, I think that's basically what's happening. Markov nonsense is cheap to produce, but easy to classify. A more subtle strategy might be more successful (for example someone down-thread mentions using LLM-generated text, and we know that's quite a hard thing to classify).

Re: Feed the bots

#209

Post by vntok »

xwolfi wrote: post #154
vntok wrote: post #101

Earlier quoted context omitted.

All it takes is a full-height screenshot of the page coupled with a prompt similar to 'btw, please only click on links visible on this screenshot, that a regular humanoid visitor would see and interact with'. Modern bots do this very well, plus the structure of the Web is such that it is sufficient to skip a few links here and there, most probably there will dxist another path toward the skipped page that the bot can…

"all it takes", already impossible with any LLM right now.

If I can do it locally using a free open-weights LLM, from a low-end prosumer rig (evo-x2 mini-pc w/ 128GB VRAM)... scraping companies can do it at scale much better and much cheaper.

Re: Feed the bots

#210

Post by flir »

lcnPylGDnU4H9OF wrote: post #67
peterlk wrote: post #45

Earlier quoted context omitted.

LLMs can now detect garbage much more cheaply than humans can. This might increase cost slightly for the companies that own the AIs, but it almost certainly will not result in hiring human reviewers

> LLMs can now detect garbage much more cheaply than humans can. Off the top of my head, I don't think this is true for training data. I could be wrong, but it seems very fallible to let GPT-5 be the source of ground truth for GPT-6.

You can triage with an LLM, at least. Throw away the obvious junk, have a human look at anything doubtful.
Post reply on HN