What people use to prevent robots and crawlers from websites?
Ask HN: What do people use to prevent crawlers?
1–10 of 123 posts
Re: Ask HN: What do people use to prevent crawlers?
#2Can you give a little more context to the question? There are various ways but they would be dependent on the reasons in the first place.
Re: Ask HN: What do people use to prevent crawlers?
#3Re: Ask HN: What do people use to prevent crawlers?
#4For well-bahaved bots, robots.txt.
For ill-behaved ones, it depends on why you're trying to block them. Rate throttling, IP blocking, requiring login, or just gating all access to the site with HTTP Basic Auth can all work.
Re: Ask HN: What do people use to prevent crawlers?
#5A password prompt/captcha? If you do not want to get crawled, do not make it public?
Re: Ask HN: What do people use to prevent crawlers?
#6I assume you are concerned about crawlers that do not respect the robots.txt file (which is the polite way to restrict them from indexing your side, but does not provide any actual protection if crawlers chose to ignore the file). Cloudflare has a tool for doing this (now part of their core service):
https://blog.cloudflare.com/introducing-scrapeshield-discove...
There's a nice Github repo with some advice on blocking scrapers:
https://github.com/JonasCz/How-To-Prevent-Scraping
Finally, you could use a plugin in your Webserver to display a CAPTCHA to visitors from IP addresses that cause a lot of requests to your site.
There are many more strategies available (up to creating fake websites / content to lead crawlers astray), but the CAPTCHA solution is the most robust one. It will not be able to protect you against crawlers that use a large source IP pool to access your site though.
Re: Ask HN: What do people use to prevent crawlers?
#7They used to use flash, for the good or the bad.
Re: Ask HN: What do people use to prevent crawlers?
#8robots.txt
then Zip bombs.
Re: Ask HN: What do people use to prevent crawlers?
#9The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice.
You really can't protect against this unless you start making the experience of regular visitors much worse.
Re: Ask HN: What do people use to prevent crawlers?
#10[deleted]