High-performance .NET by example: Filtering bot traffic
alexandrnikitin.github.io
High-performance .NET by example: Filtering bot traffic
1–10 of 54 posts
Re: High-performance .NET by example: Filtering bot traffic
#2Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so you're only blocking well behaved ones. Plus there's thousands of browser versions out there, so there's a very good chance you're blocking some users for no reason.
The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the site or throw up a captcha like cloudflare does.
Blocking bots sounds great but it just brings Google one step closer to a monopoly. Even good bots just pretend to be people nowadays because lots of people are implementing naive site protection strategies.
Edited: to be less mean
Re: High-performance .NET by example: Filtering bot traffic
#3Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so you're only blocking well behaved ones. Plus there's thousands of browser versions out there, so there's a very good chance you're blocking some users for no reason. The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the si…
> single dumbest idea
I agree with your sentiment, but you should try to be a little more constructive.
Re: High-performance .NET by example: Filtering bot traffic
#4Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so you're only blocking well behaved ones. Plus there's thousands of browser versions out there, so there's a very good chance you're blocking some users for no reason. The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the si…
Yes, you're right. There are many ways to block robots: IP, UA, behaviour analysis. An advertising company has to have UA based filtering to be compliant with standards. However, the focus of the blog post is on performance rather on how to block bots.
Re: High-performance .NET by example: Filtering bot traffic
#5Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so you're only blocking well behaved ones. Plus there's thousands of browser versions out there, so there's a very good chance you're blocking some users for no reason. The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the si…
Plus it's baking planned obsolescence into your code. As soon as you stop updating it, it will start blocking newer browser editions and versions.
Re: High-performance .NET by example: Filtering bot traffic
#6I'd probably store cached results for Dictinary> allowed, notAllowed; where int == length of the user agent. This should probably be blazing fast as well instead of keep doing those lookups.
Re: High-performance .NET by example: Filtering bot traffic
#7I'd probably store cached results for Dictinary > allowed, notAllowed; where int == length of the user agent. This should probably be blazing fast as well instead of keep doing those lookups.
I doubt that exactly that will work. There are tens of thousands of different UAs (maybe 100K). Perhaps some kind of tiny (few CPU cache lines) cache for most popular UAs could help. But again: measure, measure, measure :)
Re: High-performance .NET by example: Filtering bot traffic
#8Rather than block on UA, just add some honeypots. An invisible link. Any bot that pulls that page gets blocked as scrapers tend to pull all links from the page and follow.
Use the robots.txt to ban the pulling of specific pages. Bots 99% of the time ignore robots, so if they pull it: block
Check how quickly pages are pulled. If passes a threshold: block
Re: High-performance .NET by example: Filtering bot traffic
#9Rather than block on UA, just add some honeypots. An invisible link. Any bot that pulls that page gets blocked as scrapers tend to pull all links from the page and follow. Use the robots.txt to ban the pulling of specific pages. Bots 99% of the time ignore robots, so if they pull it: block Check how quickly pages are pulled. If passes a threshold: block
Yes, using honeypots is one of the ways to identify bots. But that wasn't the focus of the post. I'll add some clarification.
Re: High-performance .NET by example: Filtering bot traffic
#10I'd probably store cached results for Dictinary > allowed, notAllowed; where int == length of the user agent. This should probably be blazing fast as well instead of keep doing those lookups.
A FST seems like a good fit for this problem. I believe it will be much more compact than the Aho-Corasick algorithm trie structure. Depends on the size of the dictionary.