Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The bot situation on the internet is worse than you could imagine

gladeart.com

Post reply on HN
Print view View original

171–179 of 179 posts

Re: The bot situation on the internet is worse than you could imagine

#171

Post by jwr »

bee_rider wrote: post #86
jwr wrote: post #61

An interesting and sad aspect of the war on bots and scraping that is being waged is that we are hurting ourselves in the process, too. Many tasks I'm trying to get my AI assistant to do cannot be done quickly, because sites defensively prohibit access to their content. I'm not scraping: it's my agent trying to fetch a page or two to perform a task for me (such as check pricing or availability). We need a better solu…

You aren’t scraping for the sake of training a model, but scraping the prices and availability is still scraping, right? I think some of the folks running sites would rather have you go to the site and view the items “suggested based on your shopping history” (I consider these ads, the vendors might disagree), etc. I’m more sympathetic to the people running sites than the LLM training scrapers, but these are two part…

> scraping the prices and availability is still scraping

Web browsing is scraping, too.

I am not doing anything that I myself wouldn't do, it would just take me longer. I'm not mass-scraping, training new models, etc etc. I'm just using a helper tool to do some work for me.

If you prevent that, you are effectively saying: humans have to perform the manual labor of clicking and browsing through our site, they are not allowed to be helped in any way. I don't think this is the right answer.

Re: The bot situation on the internet is worse than you could imagine

#172

Post by ranger_danger »

AugSun wrote: post #170
ranger_danger wrote: post #164

Earlier quoted context omitted.

What makes a real human?

CC payment. This is the ultimate test.

Hard disagree, it's very easy for a bot to use a credit card. And not only are card numbers often stolen, they're even given to teenagers these days, and can also be owned by businesses and exist entirely virtually... so I don't think you can assume the use of a credit card can always be tied to legitimate use by a single person.

Re: The bot situation on the internet is worse than you could imagine

#173

Post by neurostimulant »

> The IPs of these bots here actually do not come from datacenters or VPNs most of the time; the overwhelming majority come from residential and mobile networks. Asian and Indonesian countries are where nearly all of them reside.

It's really awful as an indonesian, my indonesian isp regularly got blocked by HN as well :(

Re: The bot situation on the internet is worse than you could imagine

#174

Post by neurostimulant »

Aurornis wrote: post #102
GeoAtreides wrote: post #85

Earlier quoted context omitted.

>It's less proof of work and just annoying to users, and feel good to whoever added it to their site, this is being disproved in the article posted: >And so Anubis was enabled in the tar pit at difficulty 1 (lowest setting) when requests were pouring in 24/7. Before it was enabled, it was getting several hundred-thousand requests each day. As soon as Anubis became active in there, it decreased to about 11 requests af…

The Anubis difficulty setting is (or was) so high that nobody could visit the site without leaving it open for minutes or hours.

I think that only happens when you use an old browser without native sha256 support (so it fallback to pure js implementation of sha256).

Re: The bot situation on the internet is worse than you could imagine

#175

Post by hnarn »

> NOTE: Use a VPN on these pages if you don't want your IP shown in the logs, but it won't be significant amongst the millions of others anyways

Is this supposed to be a joke? Is the author expecting users to travel back in time and use a VPN so their IP is scrubbed from logs that will get published at any time, because that's something the author just obviously has the right to do?

> The EDPB explicitly identifies IP addresses as being personal data due to their ability to identify individual data subjects.[1]

Dickhead.

[1]: https://techgdpr.com/blog/is-an-ip-address-considered-person...

Re: The bot situation on the internet is worse than you could imagine

#176

Post by rchaud »

ranger_danger wrote: post #172
AugSun wrote: post #170

Earlier quoted context omitted.

CC payment. This is the ultimate test.

Hard disagree, it's very easy for a bot to use a credit card. And not only are card numbers often stolen, they're even given to teenagers these days, and can also be owned by businesses and exist entirely virtually... so I don't think you can assume the use of a credit card can always be tied to legitimate use by a single person.

Companies would offer all-you-can-DDoS plans at $20/bot per month if they could. Bots are only a problem to them because they prevent legitimate customers from handing over their credit card.

Re: The bot situation on the internet is worse than you could imagine

#177

Post by rekabis »

lifeisstillgood wrote: post #157
rekabis wrote: post #147

Earlier quoted context omitted.

> The problem here is that pretty much every part of modern life has been government and capitalism riding in the same cart - from cities installing electric power stations 100 years ago, to roads and inventions like the transistor and internet itself was government and private capital working towards common goals. And here is your argument’s fatal flaw: mistaking public works , that benefits the commons and society…

Just a sec - you said: “”” Remember: when government and capitalism rides in the same cart, it is called corporatism, and is the basis of Fascism. ””” The implication being when gov and big business get together bad things do happen. My point is they can happen yes, but also same players can make good things happen - it depends on the players (and regulatory and reporting and voting and and and ) Government tracking…

> Just a sec - you said: “”” Remember: when government and capitalism rides in the same cart, it is called corporatism, and is the basis of Fascism. ”””

>The implication being when gov and big business get together bad things do happen. My point is they can happen yes, but also same players can make good things happen - it depends on the players (and regulatory and reporting and voting and and and )

I think you need to go back and take some elementary courses on economics and how governments operate.

Public works are not “government and corporations in the same cart”. If that’s what you are coming to, that only demonstrates your ignorance bleeding out all over the place.

Re: The bot situation on the internet is worse than you could imagine

#178

Post by Saris »

lm411 wrote: post #106

AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW. I was hit with a pretty substantial botnet "distributed scraping" attack yesterday. - About 400,000 different IP addresses over about 3 hours - Mostly residential IP addresses - Valid and unique user agents and referrers - Each IP address would make only a few requests with a long delay in between requests It would hit the se…

What I don't understand is why a bot/scraper needs to load every page and image multiple times in the same hour or whatever session it's doing on my site. If I have say 10 pages and 100 images, surely 110 requests should be all it needs to load everything.

Re: The bot situation on the internet is worse than you could imagine

#179

Post by but_noah »

LeoPanthera wrote: post #25

Is Anubus being set to difficulty 8 on this page supposed to be a joke? I gave up after about 20 seconds.

The core problem isn't Proof-of-Work itself — it's difficulty calibration.

PoW for page protection (every page load) is fundamentally different from PoW for form submission (one-time action). Anubis at difficulty 8 is asking browsers to find 8 leading zero bytes — that's billions of hashes on average.

For form spam protection, a ~200ms solve time in a WebWorker is enough to make bot operations uneconomical at scale (10,000 forms = 10,000 unique computations) while being invisible to humans. That's difficulty 2-3 depending on device.

For protecting every page view like Anubis does, the cost-benefit math is different and much harder to get right, because you're taxing every visitor on every request — not just on submit.

Post reply on HN