Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The bot situation on the internet is worse than you could imagine

gladeart.com

Post reply on HN
Print view View original

81–90 of 179 posts

Re: The bot situation on the internet is worse than you could imagine

#81

Post by bob1029 »

> safari can't open the page

What is the point of these anti bot measures if organic HN traffic can nuke your site regardless? If this is about protecting information from being acquired by undesirable parties, then this site is currently operating in the most ideal way possible.

The information will eventually be ripped out. You cannot defeat an army with direct access to TSMC's wafer start budget and Microsoft's cloud infrastructure. I would find a different hill to die on. This is exactly like the cookie banners. No one is winning anything here. Publishing information to the public internet is a binary decision. If you need to control access, you do what Netflix and countless others have done. You can't have it both ways.

Re: The bot situation on the internet is worse than you could imagine

#82

Post by throw10920 »

yborg wrote: post #65
drum55 wrote: post #38

Earlier quoted context omitted.

Ironically I used a LLM to write a bypass for this ridiculous tool, doing hashing in a browser makes no sense, Claude's very bad implementation of it in C does tens of megahash a second and passes all of the challenges nearly instantly. It took about 5 minutes for Claude to write that, and it's not even a particularly fast implementation, but it beats the pants off doing string comparisons for every loop in JavaScrip…

Maybe post your brilliant solution to commercial companies with hundreds of millions in funding unrestrained bot scraping the Internet for AI training instead of complaining about people desperate to rein it in as individuals.

This is fallacious and extremely disrespectful (or even malicious?). You don't have to propose a way to fix a broken thing to point out that it's broken.

Normal and sane people understand this intuitively. If someone goes to a mechanic because their car is broken and the mechanic says "well, if you can tell that you car is broken, then you should be able to figure out how to fix it" - that mechanic would be universally hated and go out of business in months. Same thing for a customer complaining about a dish made for them in a restaurant, or a user pointing out a bug in a piece of software.

Re: The bot situation on the internet is worse than you could imagine

#83

Post by siva7 »

MeetingsBrowser wrote: post #72
siva7 wrote: post #57

So the elephant in the room: How much of HN is bot generated? Those who know have every incentive not to share and those who don't have no way to figure it out. At this point i have to assume that every new account is a bot

The article is about automated web scraping, not bots writing content.

The commenters here don't care what the article is about when they can't access the article and the much more concerning question not about web scraping is.

Re: The bot situation on the internet is worse than you could imagine

#84

Post by Retr0id »

drum55 wrote: post #56
Retr0id wrote: post #52

Earlier quoted context omitted.

I wrote one that uses opencl: https://github.com/DavidBuchanan314/anubis_offload

Bravo, you even implemented the midstate speedup from Bitcoin, that's way more impressive.

It's not exactly rocket science heh, just baffling that the original anubis impl left an order-of-magnitude speedup on the table.

Re: The bot situation on the internet is worse than you could imagine

#85

Post by GeoAtreides »

drum55 wrote: post #38
mxmlnkn wrote: post #30

Earlier quoted context omitted.

After 2 minutes at 150 kHashes on mobile, I finally see the first pixel of the progress bar filling up. Seems like it will take hours or a day to finish. Some estimate would have been nice.

Ironically I used a LLM to write a bypass for this ridiculous tool, doing hashing in a browser makes no sense, Claude's very bad implementation of it in C does tens of megahash a second and passes all of the challenges nearly instantly. It took about 5 minutes for Claude to write that, and it's not even a particularly fast implementation, but it beats the pants off doing string comparisons for every loop in JavaScrip…

>It's less proof of work and just annoying to users, and feel good to whoever added it to their site,

this is being disproved in the article posted:

>And so Anubis was enabled in the tar pit at difficulty 1 (lowest setting) when requests were pouring in 24/7. Before it was enabled, it was getting several hundred-thousand requests each day. As soon as Anubis became active in there, it decreased to about 11 requests after 24 hours, most just from curious humans.

apparently it does more than annoying users and making the site owner feel good (well, i suppose effective bot blocking would make the site owner feel quite good)

Re: The bot situation on the internet is worse than you could imagine

#86

Post by bee_rider »

jwr wrote: post #61

An interesting and sad aspect of the war on bots and scraping that is being waged is that we are hurting ourselves in the process, too. Many tasks I'm trying to get my AI assistant to do cannot be done quickly, because sites defensively prohibit access to their content. I'm not scraping: it's my agent trying to fetch a page or two to perform a task for me (such as check pricing or availability). We need a better solu…

You aren’t scraping for the sake of training a model, but scraping the prices and availability is still scraping, right?

I think some of the folks running sites would rather have you go to the site and view the items “suggested based on your shopping history” (I consider these ads, the vendors might disagree), etc.

I’m more sympathetic to the people running sites than the LLM training scrapers, but these are two parties in a many-party game and neither one is perfectly aligned with users.

Re: The bot situation on the internet is worse than you could imagine

#87

Post by tromp »

> let webWorkerURL = `${options.basePrefix}/.within.website/x/cmd/anubis/static/js/worker/sha256-${workerMethod}.mjs?cacheBuster=${options.version}`;

It looks like it's computing sha256 hashes. Such an ASIC friendly PoW has the downside that someone with ASICs would be able to either overwhelm the site or drive up the difficulty so high that CPUs can never get through.

Re: The bot situation on the internet is worse than you could imagine

#88

Post by jlarocco »

xboxnolifes wrote: post #79
wolvoleo wrote: post #27

Earlier quoted context omitted.

It's definitely more than enough to stop me as a human wanting to visit the site, so yeah. In that case a better solution would be to take the site down altogether.

Take down the site entirely because a couple humans get into a fit about it?

The site's down entirely anyway. The silly "proof of work" finishes only to tell me the site is down.

What a waste of time.

Re: The bot situation on the internet is worse than you could imagine

#90

Post by ghywertelling »

simonw wrote: post #74

> These bots are almost certainly scraping data for AI training; normal bad actors don't have funding for millions of unique IPs thrown at a page. They probably belong to several different companies. Perhaps they sell their scraped data to AI companies, or they are AI companies themselves. We can't tell, but we can guess since there aren't all that many large AI corporations out there. Is the theory here that OpenAI,…

https://parallel.ai/

I bet lot of companies want to provide search results to AI agents.

Post reply on HN