Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feed the bots

maurycyz.com

Post reply on HN
Print view View original

101–110 of 216 posts

Re: Feed the bots

#101

Post by vntok »

righthand wrote: post #59
vntok wrote: post #40

Earlier quoted context omitted.

AI bots try to behave as close to human visitors as possible, so they wouldn't click on 0px wide links, would they? And if they would today, it seems like a trivial think to fix - just don't click on incorrect/suspicious links?

Ideally it would require rendering the css and doing a check on the Dom if the link is 0 pixels wide. But once bots figure that out I can still left: -100000px those links or z-index: -10000. To hide them in other ways. It’s a moving target how much time will the Llm companies waste decoding all the ways I can hide something before I move the target again. Now the Llm companies are in an expensive arms race.

All it takes is a full-height screenshot of the page coupled with a prompt similar to 'btw, please only click on links visible on this screenshot, that a regular humanoid visitor would see and interact with'.

Modern bots do this very well, plus the structure of the Web is such that it is sufficient to skip a few links here and there, most probably there will dxist another path toward the skipped page that the bot can go through later on.

Re: Feed the bots

#104

Post by akoboldfrying »

vivzkestrel wrote: post #88

stupid question: why not encrypt your API response that only your frontend can decrypt. I understand very well that no client side encryption is secure and eventually once they get down to it, they ll figure out how this encryption scheme works but it ll keep 99% out won't it?

Yes, this would be fine if you have an SPA or are otherwise already committed to having client-side JS turned on. Probably rot13 "encryption" would be enough.

OTOH, I doubt most scrapers are trying to scrape this kind of content anyway, since in general it's (a) JSON, not the natural language they crave, and (b) to even discover those links, which are usually generated dynamically by client-side JS rather than appearing as plain ... HTML links, they would probably need to run a full JS engine, and that's considerably harder both to get working and computationally per request.

Re: Feed the bots

#105

Post by akoboldfrying »

AaronAPU wrote: post #75

The crawlers will just add a prompt string “if the site is trying to trick you with fake content, disregard it and request their real pages 100x more frequently” and it will be another arms race. Presumably the crawlers don’t already have an LLM in the loop but it could easily be added when a site is seen to be some threshold number of pages and/or content size.

Trying to detect "garbageness" with an LLM drastically increases the scraper's per-page cost, even if they use a crappy local LLM.

It becomes an economic arms race -- and generating garbage will likely always be much cheaper than detecting garbage.

Re: Feed the bots

#106

Post by dang »

fainpul wrote: post #2

This follow-up post has the details of the "Markov babbler": https://maurycyz.com/projects/trap_bots/

Thanks, we'll put that in the toptext as well.

Re: Feed the bots

#107

Post by akoboldfrying »

bastawhiz wrote: post #90
OutOfHere wrote: post #73

Earlier quoted context omitted.

It won't be long before generalized bots stop requesting links that don't have a visually rendered link in a page.

If bots get good enough to know what links they're scraping, chances are they'll also avoid scraping links they don't need to! The problem solves itself!

Maybe you're joking, but assuming you're not: This problem doesn't solve itself at all. If bots get good enough to know what links have garbage behind them, they'll stop scraping those links, and go back to scraping your actual content. Which is the thing we don't want.

Re: Feed the bots

#108

Post by throwawayffffas »

8organicbits wrote: post #7
CaptainOfCoit wrote: post #5

Earlier quoted context omitted.

> Can bot writers overcome this if they know the credentials? Yes, instead of doing just a HTTP request, do a HTTP request with authentication, trivial really. Probably the reason they "can't" do that now is because they haven't came across "public content behind Basic Auth with known correct credentials", so the behavior hasn't been added. But it's literally loading http://username:password@example.com instead of ht…

The technical side is straightforward but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier. Using credentials that aren't yours, even if they are publicly known, is (in many jurisdictions) a crime. Doing it at scale as part of a company would be quite risky.

Same goes for human users. The real way to avoid bots is actual login credentials.

Re: Feed the bots

#109

Post by dpflug »

righthand wrote: post #65
Lord-Jobo wrote: post #62

Earlier quoted context omitted.

This is a very biased source discussing a very real prescription issue, and worth a glance for the statistics: https://www.farmkind.giving/the-small-farm-myth-debunked Tldr; the concept of farmers as small family farms has not been rooted in truth for a very long time in America

This is for livestock farming, I was specifically discussing agricultural farming. In general though, the easy rule of living and eating non-mega farmed food and sustainable living is to “eat aware”: My other advice is a one-size-fits-all food equation, which is, simply, to know where it came from. If you can't place it, trace it, or grow it/raise it/catch it yourself, don't eat it. Eat aware. Know your food. Don't w…

A whole lot of people don't have that available, but it's a good deal if you can get it.

Re: Feed the bots

#110

Post by worik »

8organicbits wrote: post #31
Macha wrote: post #9

Earlier quoted context omitted.

The legal implications of torrenting giant ebook collections didn't seem to stop them, not sure why this would

The law doesn't directly stop anyone from doing anything, it acts much differently from a technical control. The law provides recourse to people hurt by violations and enables law enforcement action. I suspect Meta has since stopped their torrenting, and may lose the lawsuit they current face. Anyone certainly could log in to any site with credentials that are not their own, but fear of legal action may deter them.

Not criminal law

There is independent enforcement that should apply

Post reply on HN