Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feed the bots

maurycyz.com

Post reply on HN
Print view View original

81–90 of 216 posts

Re: Feed the bots

#81

Post by eviks »

lolpython wrote: post #79
eviks wrote: post #36

How does this help protect the regular non-garbage pages from the bots?

The follow on post explains: > You don’t really need any bot detection: just linking to the garbage from your main website will do. Because each page links to five more garbage pages, the crawler’s queue will quickly fill up with an exponential amount of garbage until it has no time left to crawl your real site. From: https://maurycyz.com/projects/trap_bots/

Thanks, I thought that these are prioritized, so while the garbage links might fill up the queue, they'd do so only after all real links are visited, so the server load is the same. But of course, not all/most bots might be configured this way.

> If a link is posted somewhere, the bots will know it exists,

Re: Feed the bots

#82

Post by akoboldfrying »

blibble wrote: post #46

if you want to be really sneaky make it so the web doesn't start off infinite because as infinite site that has appeared out of nowhere will quickly be noticed and blocked start it off small, and grow it by a few pages every day and the existing pages should stay 99% the same between crawls to gain reputation

Good thinking.

One way to keep things mostly the same without having to store any of it yourself:

1. Use an RNG seeded from the request URL itself to generate each page. This is already enough for an unchanging static site of finite it infinite size.

2. With each word the generator outputs, generate a random number between, say, 0 and 1000. On day i, replace the about-to-be-output word with a link if this random number is between 0 and i. This way, every day roughly 0.1% of words will turn into links, with the rest of the text remaining stable over time.

Re: Feed the bots

#83

Post by renegat0x0 »

I run something I call an "ethical crawler". It’s designed to avoid being a burden to websites - it makes requests very infrequently. Crawling the internet reliably has become increasingly difficult, as more and more content is protected or blocked. It’s especially frustrating when RSS feeds are inaccessible to bots.

404 definitely are not a problem for me. My crawler tests different mechanisms and browser headers while exploring the web.

My scraping mechanism:

https://github.com/rumca-js/crawler-buddy

Web crawler / RSS reader

https://github.com/rumca-js/Django-link-archive

Re: Feed the bots

#84

Post by blackhaj7 »

wodenokoto wrote: post #68
blackhaj7 wrote: post #66

Can someone explain how this works? Surely the bots are still hitting the pages they were hitting before but now they also hit the garbage pages too?

In authors setup, sending Markova generated garbage is much lighter on resources than sending static pages. Only bots will continue to follow links to the next piece of garbage and thus he traps bots in garbage. No need to detect bots, they reveal themselves. But yes, all bots start out on an actual page.

Thanks for the explanation!

Re: Feed the bots

#85

Post by iberator »

goodthink wrote: post #4

I have yet to see any bots figure out how to get past the Basic Auth protecting all links on my (zero traffic) website. Of course, any user following a link will be stopped by the same login dialog (I display the credentials on the home page). The solution is to make the secrets public. ALL websites could implement the same User/Pass credentials: User: nobots Pass: nobots Can bot writers overcome this if they know th…

[flagged]

Re: Feed the bots

#86

Post by vivzkestrel »

renegat0x0 wrote: post #83

I run something I call an "ethical crawler". It’s designed to avoid being a burden to websites - it makes requests very infrequently. Crawling the internet reliably has become increasingly difficult, as more and more content is protected or blocked. It’s especially frustrating when RSS feeds are inaccessible to bots. 404 definitely are not a problem for me. My crawler tests different mechanisms and browser headers wh…

your requirements.txt lists feedparser but where are you actually using it? https://github.com/search?q=repo%3Arumca-js%2FDjango-link-ar...

Re: Feed the bots

#87

Post by lolpython »

eviks wrote: post #81
lolpython wrote: post #79

Earlier quoted context omitted.

The follow on post explains: > You don’t really need any bot detection: just linking to the garbage from your main website will do. Because each page links to five more garbage pages, the crawler’s queue will quickly fill up with an exponential amount of garbage until it has no time left to crawl your real site. From: https://maurycyz.com/projects/trap_bots/

Thanks, I thought that these are prioritized, so while the garbage links might fill up the queue, they'd do so only after all real links are visited, so the server load is the same. But of course, not all/most bots might be configured this way. > If a link is posted somewhere, the bots will know it exists,

How would the links be prioritized? If the bots goal is to crawl all content would they have prioritization built-in?

Re: Feed the bots

#88

Post by vivzkestrel »

stupid question: why not encrypt your API response that only your frontend can decrypt. I understand very well that no client side encryption is secure and eventually once they get down to it, they ll figure out how this encryption scheme works but it ll keep 99% out won't it?

Re: Feed the bots

#89

Post by goodthink »

iberator wrote: post #85
goodthink wrote: post #4

I have yet to see any bots figure out how to get past the Basic Auth protecting all links on my (zero traffic) website. Of course, any user following a link will be stopped by the same login dialog (I display the credentials on the home page). The solution is to make the secrets public. ALL websites could implement the same User/Pass credentials: User: nobots Pass: nobots Can bot writers overcome this if they know th…

[flagged]

Three score and seven years old. How old are you? Good insane or bad? lol I can't infer it from the comment. Here are links to my 3D, multiuser, "coin-toss as a service" apps: https://chalculator.com/gaas/?world=cutcards https://chalculator.com/gaas/?world=diceroll User: croquet Pas: yadayadayada

Let the bot scraping begin.

(These were the impetus for the BA strategy. Some of the assets are large. And they were getting downloaded A LOT. Not anymore.)

Re: Feed the bots

#90

Post by bastawhiz »

OutOfHere wrote: post #73
bastawhiz wrote: post #34

Earlier quoted context omitted.

You don't need to classify bots. Bots will follow any link they find. Hide links on your pages and eventually every bot will greedily find itself in an endless labyrinth of slop.

It won't be long before generalized bots stop requesting links that don't have a visually rendered link in a page.

If bots get good enough to know what links they're scraping, chances are they'll also avoid scraping links they don't need to! The problem solves itself!
Post reply on HN