11–20 of 54 posts
I'd probably store cached results for Dictinary > allowed, notAllowed; where int == length of the user agent. This should probably be blazing fast as well instead of keep doing those lookups.
A FST seems like a good fit for this problem. I believe it will be much more compact than the Aho-Corasick algorithm trie structure. Depends on the size of the dictionary.
I'd probably store cached results for Dictinary > allowed, notAllowed; where int == length of the user agent. This should probably be blazing fast as well instead of keep doing those lookups.
I doubt that exactly that will work. There are tens of thousands of different UAs (maybe 100K). Perhaps some kind of tiny (few CPU cache lines) cache for most popular UAs could help. But again: measure, measure, measure :)
Thanks for sharing!
Earlier quoted context omitted.
A FST seems like a good fit for this problem. I believe it will be much more compact than the Aho-Corasick algorithm trie structure. Depends on the size of the dictionary.
I have that feeling too. But I found it harder to implement, especially with fallback failure references.
Also you may find this talk useful [3] (Particularly slide 11).
Great write up by the way. Really thorough on the benchmarking!
[1] https://lucene.apache.org/core/4_1_0/core/org/apache/lucene/...
[2] http://www.openfst.org/twiki/bin/view/FST/WebHome
[3] https://www.slideshare.net/lucenerevolution/text-tagging-wit...
Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so you're only blocking well behaved ones. Plus there's thousands of browser versions out there, so there's a very good chance you're blocking some users for no reason. The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the si…
If I was writing a bot, I would set user agent to some well known and very popular value, i.e. newest Chrome on Windows, or something like that.
Rather than block on UA, just add some honeypots. An invisible link. Any bot that pulls that page gets blocked as scrapers tend to pull all links from the page and follow. Use the robots.txt to ban the pulling of specific pages. Bots 99% of the time ignore robots, so if they pull it: block Check how quickly pages are pulled. If passes a threshold: block
Excellent post showing how to correctly improve code w.r.t. performance using the scientific method: hypothesis, measuring the baseline, change, measuring effect with real tools, real code. Thanks for sharing!
Meaning, yes, the OP got impressive performance improvements but the code is also completely unreadable and utilises unsafe code sections which could expose you to security problems/memory leaks/memory corruption. Not to mention they've recreated and will need to maintain an in-house version of the Dictionary class.
Their first optimisation (from Enumerator to List and Any() to Count()) are something every codebase could use. Most of their other optimisations make the code a maintenance minefield.
Plus programmers are expensive. Hardware is cheap. Why spent time on harder code to write that's also harder to maintain in the medium to long term when instead you could just throw money at hardware and call it a day? Just food for thought, not really a criticism in and of itself.
PS - Please don't take this post too seriously. I am not really being critical, just playing devil's advocate. I actually enjoyed the linked article a lot.
Excellent post showing how to correctly improve code w.r.t. performance using the scientific method: hypothesis, measuring the baseline, change, measuring effect with real tools, real code. Thanks for sharing!
I also found it an interesting post in that it kind of inadvertently proves that for most situations you shouldn't optimise to this extent. Meaning, yes, the OP got impressive performance improvements but the code is also completely unreadable and utilises unsafe code sections which could expose you to security problems/memory leaks/memory corruption. Not to mention they've recreated and will need to maintain an in-h…
It seems like most git guis are pretty commit-focused. I'm not sure of any way to do it in the command line (though it must be possible) but it would be nice to highlight a section of code in your IDE and have git give you a history of just those lines (or that function) as far back as you want to go.