Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feed the bots

maurycyz.com

Post reply on HN
Print view View original

151–160 of 216 posts

Re: Feed the bots

#151

Post by trenchpilgrim »

XenophileJKO wrote: post #147
trenchpilgrim wrote: post #146

Earlier quoted context omitted.

"I'm going to hammer your site with requests, and if I use the information I receive to cause harm to a third party, it's YOUR FAULT" is an absolutely ludicrous take.

The scrappers by violating your wishes are doing something they shouldn't. My comment is not commenting about that. What I said doesn't mean the scrapper is any less wrong. I'm basically saying 2 wrongs don't make a right here. Trying to harm their system which might transitively harm someone using their system is unethical from my viewpoint.

So you're suggesting as a website operator I should do nothing to resist and pay a large web hosting bill so that a company I've never heard of should benefit? That is more directly harmful than this hypothetical third harm. What about my right to defend myself and my property?

Re: Feed the bots

#152

Post by NotATest22 »

XenophileJKO wrote: post #150
_vertigo wrote: post #145

Earlier quoted context omitted.

Please. Are you implying we need AI to the same degree we need clean water? Your chemicals in river analogy only works if there were also a giant company straight out of “The Lorax” siphoning off all of the water in the river.. and further, the chemicals would have to be harmless to humans but would cause the company’s machines to break down so they couldn’t make any more thneeds.

The problem is: 1. The machines won't "break", at best you slightly increase when they answer something with incorrect information. 2. People are starting to rely on that information, so when 'transformed" your harmless chemical are now potentially poison. Knowing this is possible, it (again "to me") becomes highly un-ethical.

The onus to produce correct information is on the LLM producer. Even if its not poisoned information it may still be wrong. The fact that LLM producers are releasing a product that is producing information that is not verified is not a bloggers fault.

Re: Feed the bots

#153

Post by XenophileJKO »

trenchpilgrim wrote: post #151
XenophileJKO wrote: post #147

Earlier quoted context omitted.

The scrappers by violating your wishes are doing something they shouldn't. My comment is not commenting about that. What I said doesn't mean the scrapper is any less wrong. I'm basically saying 2 wrongs don't make a right here. Trying to harm their system which might transitively harm someone using their system is unethical from my viewpoint.

So you're suggesting as a website operator I should do nothing to resist and pay a large web hosting bill so that a company I've never heard of should benefit? That is more directly harmful than this hypothetical third harm. What about my right to defend myself and my property?

You should block them, that is the ethical option.

Re: Feed the bots

#154

Post by xwolfi »

vntok wrote: post #101
righthand wrote: post #59

Earlier quoted context omitted.

Ideally it would require rendering the css and doing a check on the Dom if the link is 0 pixels wide. But once bots figure that out I can still left: -100000px those links or z-index: -10000. To hide them in other ways. It’s a moving target how much time will the Llm companies waste decoding all the ways I can hide something before I move the target again. Now the Llm companies are in an expensive arms race.

All it takes is a full-height screenshot of the page coupled with a prompt similar to 'btw, please only click on links visible on this screenshot, that a regular humanoid visitor would see and interact with'. Modern bots do this very well, plus the structure of the Web is such that it is sufficient to skip a few links here and there, most probably there will dxist another path toward the skipped page that the bot can…

"all it takes", already impossible with any LLM right now.

Re: Feed the bots

#155

Post by ElectroBuffoon »

markus_zhang wrote: post #61
peterlk wrote: post #45

Earlier quoted context omitted.

LLMs can now detect garbage much more cheaply than humans can. This might increase cost slightly for the companies that own the AIs, but it almost certainly will not result in hiring human reviewers

What about garbage that are difficult to tell from truth? For example, say I have an AD&D website, how does AI tell whether a piece of FR history is canon or not? Yeah I know it's a bit extreme, but you get the idea.

If the same garbage is repeated enough all over the net, the AIs will suffer brain rot. GIGO and https://news.ycombinator.com/item?id=45656223

Next step will be to mask the real information with typ0canno. Or parts of the text, otherwise search engines will fail miserably. Also squirrel anywhere so dogs look in the other direction. Up.

Imagine filtering the meaty parts with something like /usr/games/rasterman:

> what about garbage thta are dififult to tell from truth?

> for example.. say i have an ad&d website.. how does ai etll whether a piece of fr history is canon ro not? yeah ik now it's a bit etreme.. but u gewt teh idea...

or /usr/games/scramble:

> Waht aobut ggaabre taht are dficiuflt to tlel form ttruh?

> For eapxlme, say I hvae an AD&D wisbete, how deos AI tlel wthheer a pciee of FR hsiotry is caonn or not? Yaeh I konw it's a bit emxetre, but you get the ieda.

Sadly punny humans will have a harder time decyphering the mess and trying to get the silly references. But that is a sacrifice Titans are willing to make for their own good.

ElectroBuffoon over. bttzzzz

Re: Feed the bots

#156

Post by dilyevsky »

markus_zhang wrote: post #41

I have always recommended this strategy: flood the AI bots with garbage that looks like authentic information so that they need actual humans to filter the information. Make sure that every site does this so they get more garbage than real stuffs. Hike up the proportion so that even ordinary people eventually figure out that using these AI products has more harm than use because it just produces garbage. I just don't…

LLMs already train on mostly garbage - you are just wasting your time. Same as talking to spam callers.

Re: Feed the bots

#157

Post by csomar »

mcdeltat wrote: post #123

Maybe a dumb question but what exactly is wrong with banning the IPs? Even if the bots get more IPs over time, surely storing a list of bans is cheaper than serving content? Is the worry that the bots will eventually cycle through so many IP ranges that you end up blocking legit users?

They are usually using residential IPs through SOCK5. I am not sure how they are getting these residential IPs but it is definitively suspicious.

So by blocking these IPs, you are blocking your users. (ie: in many coffeshops, I get the "IP Blocked" banner, my guess is that they are running software on unsuspecting users to route this traffic).

Re: Feed the bots

#158

Post by cryptonector »

gridspy wrote: post #124
kelnos wrote: post #121

Earlier quoted context omitted.

Not sure if I agree with you on the thread exhaustion issue. The client can still send a flood of correctly-formed requests; the reverse proxy will pass them all through. As I said above, yes, the fact that babble processes requests so quickly would make this harder, but you could still end up with (tens of?) thousands of concurrent requests if someone is really determined to mess with you. A solution could be to lim…

Thread exhaustion attack 1. Start connections to a server 2. Hold connections open 3. Do nothing else Server 1. Incoming connection. assign a thread. 2. Wait for request 3. Serve request 4. Close connection and thread / return to threadpool Solution: Use a reverse proxy to handle the incoming connections. Typical reverse proxies such as nginx use event-based polling not a per-connection thread so they are immune to t…

The way you deal with this is that you write the server to be async I/O based with NPROC threads, not a thread-per-client design, and then you can use CPS for the business logic, but in this case it's so trivial... You can probably get by with just a handful of bytes of memory pressure per client in the app + whatever the per-client TCB is for the TCP connection for a total of less than 200 bytes per client.

Re: Feed the bots

#159

Post by marginalia_nu »

XenophileJKO wrote: post #153
trenchpilgrim wrote: post #151

Earlier quoted context omitted.

So you're suggesting as a website operator I should do nothing to resist and pay a large web hosting bill so that a company I've never heard of should benefit? That is more directly harmful than this hypothetical third harm. What about my right to defend myself and my property?

You should block them, that is the ethical option.

If that worked this wouldn't be a discussion.

Most of these misbehaved crawlers are either cloud hosted (with tens of thousands of IPs), using residential proxies (with tens of thousands of IPs) or straight up using a botnet (again with tens of thousands of IPs). None respect robots.txt and precious few even provide an identifiable user-agent string.

Re: Feed the bots

#160

Post by marginalia_nu »

andai wrote: post #119

I am confused where this traffic is coming from. OP says it's from well funded AI companies. But there are not such a large number of those? Why would they need to scrape the same pages over and over? Or is the scraping happening in real time due to the web search features in AI apps? (Cheaper to load the same page again than to cache it?)

Crawlers are pretty hard to build, they have an insane number of corner cases they need to deal with if you want them to perform well AND be perceived as respectful, and crawlers (if you go that route) find themselves among the harder problems in distributed computing, with a huge shared mutable state and some very complex shared timers.

If you're in a hurry to race to the market, it's very likely you'll run into these issues and find yourself tempted to cut corners, and unfortunately, with nearly unbounded cloud spend, cutting corners in a large scale crawler operation can very believably cause major disruption all over the web.

Post reply on HN