You have to get more aggressive unfortunately which may sometimes block real users but do the following: - Setup captcha or just block users from certain countries if you know where your traffic comes from. This can sometimes create issues for your users on VPN but then you have to make the call depending on how many of your users may be using VPN etc. At the minimum, add a captcha. - Create more Page rules in cloudf…
Ask HN: Best way to stop bot traffic?
21–30 of 32 posts
Re: Ask HN: Best way to stop bot traffic?
#22I'd go against the "just increase the cf strictness" advice. It's counting on cf basically doing something magic and hoping to not about real users - and that's not really possible. 1. Why do you want to stop bots? Are they actually overloading your resources, or are they just noisy in the logs. If you can easily handle the traffic, maybe find a way to filter the logs better. 2. How do you know they're bots? If they'…
The purpose of this thread is to gather ideas and experiences. A silver bullet would be great, but since it's not realistic, all ideas are more than welcome.
Re: Ask HN: Best way to stop bot traffic?
#23Try increasing the security level in CF. You could also activate Firewall rules for certain countries that are not relevant for your site, make sure to not apply it to known bots (googlebot etc).
Re: Ask HN: Best way to stop bot traffic?
#24If you're interested in a non-Cloudflare solution, we have developed a service called Gatekeeper, and we'd really like to get your thoughts on whether it might suit your needs: https://www.nettoolkit.com/gatekeeper/about Essentially, Gatekeeper is a rules-engine with a fancy UI that allows you to craft policies specific to your site and the traffic that is visiting your site. For example, you can say "Allow Googlebot…
Re: Ask HN: Best way to stop bot traffic?
#25I use cloudflare and only allow paths I know. i.e /blog* /app/* I block everything else That kills most of it.
Re: Ask HN: Best way to stop bot traffic?
#26Man, if the state of the art is to suggest something Cloudflare related, that's a really sad state of affairs.
Re: Ask HN: Best way to stop bot traffic?
#27Man, if the state of the art is to suggest something Cloudflare related, that's a really sad state of affairs.
Re: Ask HN: Best way to stop bot traffic?
#28Man, if the state of the art is to suggest something Cloudflare related, that's a really sad state of affairs.
OP even said they gave up on cloudflare but people are trying to upsell him on the higher tiers through this hn post!
Re: Ask HN: Best way to stop bot traffic?
#29If you're interested in a non-Cloudflare solution, we have developed a service called Gatekeeper, and we'd really like to get your thoughts on whether it might suit your needs: https://www.nettoolkit.com/gatekeeper/about Essentially, Gatekeeper is a rules-engine with a fancy UI that allows you to craft policies specific to your site and the traffic that is visiting your site. For example, you can say "Allow Googlebot…
How do you differ from Cloudflare?
Gatekeeper also allows you to look over previous traffic patterns and adjust your custom rules. Philosophically, our assumption is that website operators know their sites the best, and know what content they are sensitive about and Gatekeeper is a tool to help review traffic and express custom rules. I think Cloudflare is a better fit for operators who don't want to spend time fiddling with various very granular knobs to optimize for their specific site and their specific concerns.
Gatekeeper can also be used for other purposes. For example, the idea behind Gatekeeper originated because we have a site where we want to make information freely available to humans, but not to scrapers. However, Gatekeeper could also be used to upsell users who frequent your site (e.g. "We noticed you've viewed 10 articles in the past week, please subscribe!").
If you're considering custom logic like https://news.ycombinator.com/item?id=33160679 , Gatekeeper already has a rules engine so that you specify such policies without having to write your own infrastructure.
I'd be happy to chat further to learn more about what you're looking for and to see if Gatekeeper might be able to help. Feel free to reach out via email (in profile) if you'd like to talk more.
Re: Ask HN: Best way to stop bot traffic?
#30Most bots don't bother setting cookies, or downloading CSS. Exploit this by including a dummy CSS file on your site that, on the backend, stores the visitor's IP in some kind of database, or sets a cookie. If you get multiple visits from an IP that never hit the CSS file, you can be reasonably confident the user is not legit. You need to be careful about not blocking good bots though. Do a reverse DNS lookup before a…