Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feed the bots

maurycyz.com

Post reply on HN
Print view View original

51–60 of 216 posts

Re: Feed the bots

#51

Post by righthand »

zkmon wrote: post #24

Really cool. Reminds me of farmers of some third world countries. Completely ignored by government, exploited by commission brokers, farmers now use all sorts of tricks, including coloring and faking their farm produce, without regard for health hazards to consumers. The city dwellers who thought they have gamed the system through high education, jobs and slick-talk, have to consume whatever is served to them by the…

The agricultural farmers did it to themselves, many are very wealthy already. Anything corporate America has taken over is because the farmers didn’t want to do the maintenance work. So they sell out to big corporations who will make it easier.

Same as any other consumer using Meta products. You sell out because it’s easier to network that way.

I am the son of a farmer.

Edit: added disclosure at the bottom and clarified as agricultural farming

Re: Feed the bots

#54

Post by YouAreWRONGtoo »

stubish wrote: post #28
OutOfHere wrote: post #17

The user's approach would work only if bots can accurately even be classified, but this is impossible. The end result is that the action is user's site is now nothing but markov garbage. Not only will bots desert it but humans will too.

The traditional approach is a link to the tarpit that the bots can see but humans can't, say using CSS to render it 0 pixels in size.

[dead]

Re: Feed the bots

#55

Post by TekMol »

How about adding some image with a public http logger url like

https://ih879.requestcatcher.com/test

to each of the nonsense pages, so we can see an endless flood of funny requests at

https://ih879.requestcatcher.com

?

I'm not sure requestcatcher is a good one, it's just the first one that came up when I googled. But I guess there are many such services, or one could also use some link shortener service with public logs.

Re: Feed the bots

#56

Post by CaptainOfCoit »

hn8726 wrote: post #49
CaptainOfCoit wrote: post #14

Earlier quoted context omitted.

> but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier If you're doing something alike to cracking then yeah. But if the credentials are right there on the landing page, and visible to the public, it's not really cracking anymore since you already know the right password before you try it, and the website that put up the basic auth is freely sharing the pa…

Otoh if, as a human, you use a known (even leaked on the website) password to "bypass the security" in order to "gain access to content you're not authorized to see", I think you'd get in trouble. I'd like if the same logic aplied to bots - implement basic (albeit weak) security and only allow access to humans. This way bots have to _hack you_ to read the content

> you use a known (even leaked on the website) password to "bypass the security" in order to "gain access to content you're not authorized to see", I think you'd get in trouble

I agree, but if someone has a website that says "This isn't the real page, go to /real.html and when authentication pops up, enter user:password", then I'd argue that is no longer "gaining access to content you're not authorized to see", the author of the page shared the credentials themselves, and acknowledged they aren't trying to hide anything, just providing a non-typical way of accessing the (for all intents and purposes, public) content.

Re: Feed the bots

#57

Post by jcynix »

vntok wrote: post #40
stubish wrote: post #28

Earlier quoted context omitted.

The traditional approach is a link to the tarpit that the bots can see but humans can't, say using CSS to render it 0 pixels in size.

AI bots try to behave as close to human visitors as possible, so they wouldn't click on 0px wide links, would they? And if they would today, it seems like a trivial think to fix - just don't click on incorrect/suspicious links?

The 0px rule would be in a separate .CSS file. I doubt that bots load .CSS files for .html files, at least I don't remember seeing this in my server logs.

And another "classic" solution is to use white link text on white background, or a font with zero width characters, all stuff which is rather unlikely to be analysed by a scraper interested primarily in text.

Re: Feed the bots

#58

Post by akoboldfrying »

morkalork wrote: post #22
CaptainOfCoit wrote: post #5

Earlier quoted context omitted.

> Can bot writers overcome this if they know the credentials? Yes, instead of doing just a HTTP request, do a HTTP request with authentication, trivial really. Probably the reason they "can't" do that now is because they haven't came across "public content behind Basic Auth with known correct credentials", so the behavior hasn't been added. But it's literally loading http://username:password@example.com instead of ht…

The bot protection on low traffic sites can be hilarious in how simple and effective it can be. Just click this checkbox. That's it. But it's not a check box matching a specific pattern provided by a well-known service, so until the bot writer inspects the site and adds the case it'll work. A browser running openai operator or whatever its called would immediately figure it out though.

> A browser running openai operator or whatever its called would immediately figure it out though.

But running that costs money, which is a disincentive. (How strong of a disincentive depends on how much it costs vs. the estimated value of a scraped page, but I think it would 100x the per-page cost at least.)

Re: Feed the bots

#59

Post by righthand »

vntok wrote: post #40
stubish wrote: post #28

Earlier quoted context omitted.

The traditional approach is a link to the tarpit that the bots can see but humans can't, say using CSS to render it 0 pixels in size.

AI bots try to behave as close to human visitors as possible, so they wouldn't click on 0px wide links, would they? And if they would today, it seems like a trivial think to fix - just don't click on incorrect/suspicious links?

Ideally it would require rendering the css and doing a check on the Dom if the link is 0 pixels wide. But once bots figure that out I can still left: -100000px those links or z-index: -10000. To hide them in other ways. It’s a moving target how much time will the Llm companies waste decoding all the ways I can hide something before I move the target again. Now the Llm companies are in an expensive arms race.

Re: Feed the bots

#60

Post by theturtlemoves »

Does this really work though? I know nothing about the inner workings of LLMs, but don't you want to break their word associations? Rather than generating "garbage" text based on which words tend to occur together and LLMs generating text based on which words it has seen together, don't you want to give them text that relates unrelated words?
Post reply on HN