Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feed the bots

maurycyz.com

Post reply on HN
Print view View original

31–40 of 216 posts

Re: Feed the bots

#31

Post by 8organicbits »

Macha wrote: post #9
8organicbits wrote: post #7

Earlier quoted context omitted.

The technical side is straightforward but the legal implications of trying passwords to try to scrape content behind authentication could pose a barrier. Using credentials that aren't yours, even if they are publicly known, is (in many jurisdictions) a crime. Doing it at scale as part of a company would be quite risky.

The legal implications of torrenting giant ebook collections didn't seem to stop them, not sure why this would

The law doesn't directly stop anyone from doing anything, it acts much differently from a technical control. The law provides recourse to people hurt by violations and enables law enforcement action. I suspect Meta has since stopped their torrenting, and may lose the lawsuit they current face. Anyone certainly could log in to any site with credentials that are not their own, but fear of legal action may deter them.

Re: Feed the bots

#32

Post by lfkdev »

goodthink wrote: post #4

I have yet to see any bots figure out how to get past the Basic Auth protecting all links on my (zero traffic) website. Of course, any user following a link will be stopped by the same login dialog (I display the credentials on the home page). The solution is to make the secrets public. ALL websites could implement the same User/Pass credentials: User: nobots Pass: nobots Can bot writers overcome this if they know th…

Not sure if I can follow you, why would credentials known by anyone stop bots?

Re: Feed the bots

#33

Post by bastawhiz »

nodja wrote: post #19

Why create the markov text server side? If the bots are running javascript just have their client generate it.

1. The bots have essentially unlimited memory and CPU. That's the cheapest part of any scraping setup.

2. You need to send the data for the Markov chain generator to the client, along with the code. This is probably bigger than the response you'd be sending anyway. (And good luck getting a bot to cache JavaScript)

3. As the author said, each request uses microseconds of CPU and just over a megabyte of RAM. This isn't taxing for anyone.

Re: Feed the bots

#34

Post by bastawhiz »

OutOfHere wrote: post #17

The user's approach would work only if bots can accurately even be classified, but this is impossible. The end result is that the action is user's site is now nothing but markov garbage. Not only will bots desert it but humans will too.

You don't need to classify bots. Bots will follow any link they find. Hide links on your pages and eventually every bot will greedily find itself in an endless labyrinth of slop.

Re: Feed the bots

#35

Post by mrweasel »

akoboldfrying wrote: post #3

My initial reaction was that running something like this is still a loss, because it probably costs you as much or more than it costs them in terms of both network bytes and CPU. But then I realised two things: 1. If they are using residential IPs, each byte of network bandwidth is probably costing them a lot more than it's costing you. Win. 2. More importantly, if this became a thing that a large fraction of all web…

We should encourage number 2. So much of the content that the AI companies are scraping is already garbage, and that's a problem. E.g. LLMs are frequently confidently wrong, but so is Reddit, who produce a large volume of trading data. We've seen a study surgesting that you can poison an LLM with very little data. Encouraging the AI companies to care about the quality of the data they are scraping could be beneficial to all.

The cost of being critical of source material might make some AI companies tank, but that seems inevitable.

Re: Feed the bots

#37

Post by vntok »

bastawhiz wrote: post #33
nodja wrote: post #19

Why create the markov text server side? If the bots are running javascript just have their client generate it.

1. The bots have essentially unlimited memory and CPU. That's the cheapest part of any scraping setup. 2. You need to send the data for the Markov chain generator to the client, along with the code. This is probably bigger than the response you'd be sending anyway. (And good luck getting a bot to cache JavaScript) 3. As the author said, each request uses microseconds of CPU and just over a megabyte of RAM. This isn't…

> 1. The bots have essentially unlimited memory and CPU. That's the cheapest part of any scraping setup.

Anyone crawling at scale would try to limit the per-request memory and CPU bounds, no? Surely you'd try to minimize resource contention at least a little bit?

Re: Feed the bots

#39

Post by codeduck »

eviks wrote: post #36

How does this help protect the regular non-garbage pages from the bots?

it does at a macroscopic level by making scraping expensive. If every "valid" page is scattered at random amongst a tarpit of recursive pages of nonsense, it becomes computationally and temporaly expensive to scrape a site for "good" data.

A single site doing this does nothing. But many sites doing this has a severe negative impact on the utility of AI scrapers - at least, until a countermeasure is developed.

Re: Feed the bots

#40

Post by vntok »

stubish wrote: post #28
OutOfHere wrote: post #17

The user's approach would work only if bots can accurately even be classified, but this is impossible. The end result is that the action is user's site is now nothing but markov garbage. Not only will bots desert it but humans will too.

The traditional approach is a link to the tarpit that the bots can see but humans can't, say using CSS to render it 0 pixels in size.

AI bots try to behave as close to human visitors as possible, so they wouldn't click on 0px wide links, would they?

And if they would today, it seems like a trivial think to fix - just don't click on incorrect/suspicious links?

Post reply on HN