Live data from Hacker News

Ask HN: Best way to stop bot traffic?

news.ycombinator.com

21–30 of 32 posts

Re: Ask HN: Best way to stop bot traffic?

#21
post #4

You have to get more aggressive unfortunately which may sometimes block real users but do the following: - Setup captcha or just block users from certain countries if you know where your traffic comes from. This can sometimes create issues for your users on VPN but then you have to make the call depending on how many of your users may be using VPN etc. At the minimum, add a captcha. - Create more Page rules in cloudf…

Blocking real users is not an option - but I can see that this would be a good setup if blocking bots were of the absolute highest priority.

Re: Ask HN: Best way to stop bot traffic?

#22
post #8

I'd go against the "just increase the cf strictness" advice. It's counting on cf basically doing something magic and hoping to not about real users - and that's not really possible. 1. Why do you want to stop bots? Are they actually overloading your resources, or are they just noisy in the logs. If you can easily handle the traffic, maybe find a way to filter the logs better. 2. How do you know they're bots? If they'…

1. Bots are more than half of my traffic right now and they don't provide any benefits except using my bandwidth and distort my statistics. 2. Strange behavior + technology 2a-c. Could be a part of a solution. 3. Good idea

The purpose of this thread is to gather ideas and experiences. A silver bullet would be great, but since it's not realistic, all ideas are more than welcome.

Re: Ask HN: Best way to stop bot traffic?

#23

Try increasing the security level in CF. You could also activate Firewall rules for certain countries that are not relevant for your site, make sure to not apply it to known bots (googlebot etc).

I am concerned about false positives - but I will weigh cons and pros. Thank you for your suggestion.

Re: Ask HN: Best way to stop bot traffic?

#24

If you're interested in a non-Cloudflare solution, we have developed a service called Gatekeeper, and we'd really like to get your thoughts on whether it might suit your needs: https://www.nettoolkit.com/gatekeeper/about Essentially, Gatekeeper is a rules-engine with a fancy UI that allows you to craft policies specific to your site and the traffic that is visiting your site. For example, you can say "Allow Googlebot…

How do you differ from Cloudflare?

Re: Ask HN: Best way to stop bot traffic?

#28

Man, if the state of the art is to suggest something Cloudflare related, that's a really sad state of affairs.

Totally agree. I’m thinking in terms and concepts like “iptables”, “ipset”, “fail2ban”, “cidr”, “asn”… while everyone else is thinking about how to change their cloudflare settings.

OP even said they gave up on cloudflare but people are trying to upsell him on the higher tiers through this hn post!

Re: Ask HN: Best way to stop bot traffic?

#29
post #24

If you're interested in a non-Cloudflare solution, we have developed a service called Gatekeeper, and we'd really like to get your thoughts on whether it might suit your needs: https://www.nettoolkit.com/gatekeeper/about Essentially, Gatekeeper is a rules-engine with a fancy UI that allows you to craft policies specific to your site and the traffic that is visiting your site. For example, you can say "Allow Googlebot…

How do you differ from Cloudflare?

In terms of implementation, one key difference is that Cloudflare sits between your servers and your users, whereas with Gatekeeper, your servers query Gatekeeper before deciding what response to send to users. One of the ramifications is that with Gatekeeper, your users never see a third-party page about verifying that they are not bots. It also means that when using Gatekeeper, you can actually debug if/when something unexpected happens.

Gatekeeper also allows you to look over previous traffic patterns and adjust your custom rules. Philosophically, our assumption is that website operators know their sites the best, and know what content they are sensitive about and Gatekeeper is a tool to help review traffic and express custom rules. I think Cloudflare is a better fit for operators who don't want to spend time fiddling with various very granular knobs to optimize for their specific site and their specific concerns.

Gatekeeper can also be used for other purposes. For example, the idea behind Gatekeeper originated because we have a site where we want to make information freely available to humans, but not to scrapers. However, Gatekeeper could also be used to upsell users who frequent your site (e.g. "We noticed you've viewed 10 articles in the past week, please subscribe!").

If you're considering custom logic like https://news.ycombinator.com/item?id=33160679 , Gatekeeper already has a rules engine so that you specify such policies without having to write your own infrastructure.

I'd be happy to chat further to learn more about what you're looking for and to see if Gatekeeper might be able to help. Feel free to reach out via email (in profile) if you'd like to talk more.

Re: Ask HN: Best way to stop bot traffic?

#30

Most bots don't bother setting cookies, or downloading CSS. Exploit this by including a dummy CSS file on your site that, on the backend, stores the visitor's IP in some kind of database, or sets a cookie. If you get multiple visits from an IP that never hit the CSS file, you can be reasonably confident the user is not legit. You need to be careful about not blocking good bots though. Do a reverse DNS lookup before a…

sounds like a great way for folks that utilize a simple curl your site to see if things are working right to get on the block list lol
Post reply on HN