Everybody says that bots put websites down, while marketing oriented folks start practicing AO (agent optimization) - to make their offerings even more available and penetrating.
Good luck banning yourself from the future.
The bot situation on the internet is worse than you could imagine
71–80 of 179 posts
Re: The bot situation on the internet is worse than you could imagine
#72So the elephant in the room: How much of HN is bot generated? Those who know have every incentive not to share and those who don't have no way to figure it out. At this point i have to assume that every new account is a bot
The article is about automated web scraping, not bots writing content.
Re: The bot situation on the internet is worse than you could imagine
#73My grad school research was on computational models of human/machine cognition, and I'm now commercializing it as a 'proof-of-human API' for bot detection, spam reduction, and identity verification. One of the mistakes people assume is that AI capability means humanness. If you know exactly where to look, you can start to identify differences between improving frontier models and human cognition. One concrete example…
How does your software work with blind people like me who use screen readers?
Your key finding is that humans process the grid as one visual scene — but that's a finding about sighted cognition.
Isn't this, like most things, a sensitivity specificity tradeoff?
How many real humans should be blocked from your system to keep the bots out?
What is the Blackstone ratio of accessibility?
Re: The bot situation on the internet is worse than you could imagine
#74> These bots are almost certainly scraping data for AI training; normal bad actors don't have funding for millions of unique IPs thrown at a page. They probably belong to several different companies. Perhaps they sell their scraped data to AI companies, or they are AI companies themselves. We can't tell, but we can guess since there aren't all that many large AI corporations out there.
Is the theory here that OpenAI, Anthropic, Gemini, xAI, Qwen, Z.ai etc are all either running bad scrapers via domestic proxies in Indonesia, or are buying data from companies that run those scrapers?
I want to know for sure. Who is paying for this activity? What does the marketplace for scraped data look like?
Re: The bot situation on the internet is worse than you could imagine
#75[dead]
Re: The bot situation on the internet is worse than you could imagine
#76Earlier quoted context omitted.
>I'm surprised at the effectiveness of simple PoW to stop practically all activity. It's even dumber than that, because by default anubis whitelists the curl user agent. curl -H "User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36" "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/diff/?id=v7.0-rc5&id2=v7.0-rc4&dt=2" Making sure…
Anubis' white lists and block rules are configurable though. The defaults are a bit silly.
The default is to allow non-Mozilla user agents so that existing (good) automation continues to work and so that people stopped threatening to burn my house down. Lovely people in the privacy community.
Re: The bot situation on the internet is worse than you could imagine
#77My grad school research was on computational models of human/machine cognition, and I'm now commercializing it as a 'proof-of-human API' for bot detection, spam reduction, and identity verification. One of the mistakes people assume is that AI capability means humanness. If you know exactly where to look, you can start to identify differences between improving frontier models and human cognition. One concrete example…
>The first problem can be reduced to a machine learning classifier. The second is an unsolved scientific problem.
I can't believe people are still using this as a generic anti-AI argument even though a decade ago people were insisting that there's no way AI can have the capabilities that frontier LLMs have today. Moreover it's unclear whether the gap even exists. Even if we take the claim that the grid pattern is some sort of fundamental constraint that AI models can't surpass, it doesn't seem too hard to work around by infilling the grids pattern and presenting the 9 images to LLMs as one image.
Re: The bot situation on the internet is worse than you could imagine
#78Employ constant faceID can deter it
Please drink verification can.
Re: The bot situation on the internet is worse than you could imagine
#79I'm surprised at the effectiveness of simple PoW to stop practically all activity. I'll implement Anubis at low difficulty for all my projects and leave a decent llms.txt referenced in my sitemap and robots.txt so LLMs can still get relevant data for my site while.keeping bad bots out. I'm getting thousands of requests from China that have really increased costs, glad it seems the fix is rather easy.
It's definitely more than enough to stop me as a human wanting to visit the site, so yeah. In that case a better solution would be to take the site down altogether.
Take down the site entirely because a couple humans get into a fit about it?
Re: The bot situation on the internet is worse than you could imagine
#80You can build some great anti-bot mechanisms with simple https://github.com/abrahamjuliot/creepjs logic. A normal user will often appear 31% or lower 'like headless score', mobile is a bit different. You'll still have trouble against sophisticated infra: https://x.com/_alexspring/status/2037968450753335617