High-performance .NET by example: Filtering bot traffic • Hacker News Board

Re: High-performance .NET by example: Filtering bot traffic

#31

Post by jackmott » Mon, Apr 24, 2017, 5:13 PM UTC

Someone1234 wrote: post #19

hdhzy wrote: post #13

Excellent post showing how to correctly improve code w.r.t. performance using the scientific method: hypothesis, measuring the baseline, change, measuring effect with real tools, real code. Thanks for sharing!

I also found it an interesting post in that it kind of inadvertently proves that for most situations you shouldn't optimise to this extent. Meaning, yes, the OP got impressive performance improvements but the code is also completely unreadable and utilises unsafe code sections which could expose you to security problems/memory leaks/memory corruption. Not to mention they've recreated and will need to maintain an in-h…

Hardware might be cheap at some scales, but what if your service gets popular?

What it latency for a single request can't be improved with more cores?

What if your product is used by consumers who may have old hardware? or phones, or watches, or laptops and they want their battery to last?

What if you consistently practiced at making high performance code, maybe then it wouldn't seem "unreadable" to you any more?

What if Linq-like higher order functions weren't slow? https://github.com/jackmott/LinqFaster

What if slow software was common today because of modern attitudes, and I wasn't seeing any increase in stability or features to show for it?

Re: High-performance .NET by example: Filtering bot traffic

#32

Post by throwasehasdwi » Mon, Apr 24, 2017, 5:22 PM UTC

jlg23 wrote: post #27

throwasehasdwi wrote: post #2

Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so you're only blocking well behaved ones. Plus there's thousands of browser versions out there, so there's a very good chance you're blocking some users for no reason. The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the si…

In theory you are right, but in reality 99% of the rogue bots are actually just some scraper tools were ignorant users changed sane defaults to "make it go faster". They usually don't have enough knowledge to even understand that they are routed into a black hole - not to speak of being able to do something about it. Disclaimer: Getting rid of those idiots^wmisguides poor souls is part of my job description.

Then why not block based on bots that ignore robots text or make requests too quickly like I suggested? Looking for sub-strings in the UA header is a hack at best.

Re: High-performance .NET by example: Filtering bot traffic

#33

Post by oblio » Mon, Apr 24, 2017, 5:37 PM UTC

doubleplusgood wrote: post #14

I did something similar with nginx, the data file from 51degrees and some lua code; each instance only handles 10-20k requests/sec so no clever optimization was needed.

Would you mind posting the Lua code?

Re: High-performance .NET by example: Filtering bot traffic

#34

Post by [deleted] » Mon, Apr 24, 2017, 5:46 PM UTC

alexandrnikitin wrote: post #4

throwasehasdwi wrote: post #2

Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so you're only blocking well behaved ones. Plus there's thousands of browser versions out there, so there's a very good chance you're blocking some users for no reason. The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the si…

Yes, you're right. There are many ways to block robots: IP, UA, behaviour analysis. An advertising company has to have UA based filtering to be compliant with standards. However, the focus of the blog post is on performance rather on how to block bots.

[deleted]

Re: High-performance .NET by example: Filtering bot traffic

#35

Post by Someone1234 » Mon, Apr 24, 2017, 5:54 PM UTC

osd wrote: post #25

Someone1234 wrote: post #19

Earlier quoted context omitted.

I also found it an interesting post in that it kind of inadvertently proves that for most situations you shouldn't optimise to this extent. Meaning, yes, the OP got impressive performance improvements but the code is also completely unreadable and utilises unsafe code sections which could expose you to security problems/memory leaks/memory corruption. Not to mention they've recreated and will need to maintain an in-h…

No, no no no. I cannot believe that you are honestly saying a 2x increase to throughput in production is something you "shouldn't take seriously" because the code isn't as readable as it was before. Programmers are expensive. Hardware is cheap. That doesn't justify completely throwing out the window any performance increasing changes just because a fresh college grad won't be able to understand what's going on within…

> I cannot believe that you are honestly saying a 2x increase to throughput in production is something you "shouldn't take seriously" because the code isn't as readable as it was before.

I've worked on large and old codebases for years. I've seen plenty of examples of where a "clever" programmer has optimised the heck out of a section of code, made it completely unmaintainable, and as a result forced a re-write (the resulting code, which was slower, was also easier to maintain and reason about).

"Production" is a meaningless rallying cry. Everything is production sooner or later. Not everything needs to be fast, although there are critical areas of a typical project that do. It is really a question of code quality relative to performance, unless performance itself is actually causing you problems. Typically these overeager performance fixes are done to code preemptively.

This is all fine for your pet and toy projects, go work on something enterprise grade. Big code means clean code. I'd take ten lines of clean easy to consume code over one "clever" line of genius.

Re: High-performance .NET by example: Filtering bot traffic

#36

Post by jlg23 » Mon, Apr 24, 2017, 6:57 PM UTC

throwasehasdwi wrote: post #32

jlg23 wrote: post #27

Earlier quoted context omitted.

In theory you are right, but in reality 99% of the rogue bots are actually just some scraper tools were ignorant users changed sane defaults to "make it go faster". They usually don't have enough knowledge to even understand that they are routed into a black hole - not to speak of being able to do something about it. Disclaimer: Getting rid of those idiots^wmisguides poor souls is part of my job description.

Then why not block based on bots that ignore robots text or make requests too quickly like I suggested? Looking for sub-strings in the UA header is a hack at best.

They all require one to keep state information on the server side and with overall little benefit. Yes, I do what you proposed, but in hindsight it was a waste of time to set that up: 99% simply don't know what they are doing and therefore easy to catch.

I don't care about this one person who knows to change the user agent sent, those are the ones you can usually talk to and they'll happily throttle their crawlers.

The other 99 are the problem - and they are a problem that can be solved pretty well by simple string matching.

Re: High-performance .NET by example: Filtering bot traffic

#37

Post by deanCommie » Mon, Apr 24, 2017, 7:02 PM UTC

jackmott wrote: post #31

Someone1234 wrote: post #19

Earlier quoted context omitted.

I also found it an interesting post in that it kind of inadvertently proves that for most situations you shouldn't optimise to this extent. Meaning, yes, the OP got impressive performance improvements but the code is also completely unreadable and utilises unsafe code sections which could expose you to security problems/memory leaks/memory corruption. Not to mention they've recreated and will need to maintain an in-h…

Hardware might be cheap at some scales, but what if your service gets popular? What it latency for a single request can't be improved with more cores? What if your product is used by consumers who may have old hardware? or phones, or watches, or laptops and they want their battery to last? What if you consistently practiced at making high performance code, maybe then it wouldn't seem "unreadable" to you any more? Wha…

> what if your service gets popular?

Then re-design for map-reduce, and scale horizontally

> What it latency for a single request can't be improved with more cores?

Then look into pre-computing and caching

> What if your product is used by consumers who may have old hardware? or phones, or watches, or laptops and they want their battery to last?

I thought we were talking about server requests? If we are, then offload this work to the server

> What if you consistently practiced at making high performance code, maybe then it wouldn't seem "unreadable" to you any more?

But it's not all about you. Unless you're working on a pet project, or you have the credibility and reputation to be the final call on a significant open source project, you might get hit by a bus tomorrow. Or, if you do a really good job, your company will need you to be a force multiplier to teach a dozen others to try to imitate you. Even if you're a "10x" programmer.

And by the way, when you optimize code THIS much, any refactoring or tweaks to new features cause your optimizations to get tossed out, and you have to start over from scratch.

> What if Linq-like higher order functions weren't slow? https://github.com/jackmott/LinqFaster

https://github.com/jackmott/LinqFaster#limitations

> What if slow software was common today because of modern attitudes, and I wasn't seeing any increase in stability or features to show for it?

Except you are, and you don't even realize it. Optimizations like this blog post matter a LOT on client software. Be it apps, or websites - anything run on the client will need this kind of attention sometimes.

But this guy is writing server software. Micro-optimizing on the server side the way he is doing is silly.

Re: High-performance .NET by example: Filtering bot traffic

#38

Post by pc86 » Mon, Apr 24, 2017, 7:29 PM UTC

jackmott wrote: post #31

Someone1234 wrote: post #19

Earlier quoted context omitted.

I also found it an interesting post in that it kind of inadvertently proves that for most situations you shouldn't optimise to this extent. Meaning, yes, the OP got impressive performance improvements but the code is also completely unreadable and utilises unsafe code sections which could expose you to security problems/memory leaks/memory corruption. Not to mention they've recreated and will need to maintain an in-h…

Hardware might be cheap at some scales, but what if your service gets popular? What it latency for a single request can't be improved with more cores? What if your product is used by consumers who may have old hardware? or phones, or watches, or laptops and they want their battery to last? What if you consistently practiced at making high performance code, maybe then it wouldn't seem "unreadable" to you any more? Wha…

I don't want to pick one single thing about your response, because I agree with a good portion of it, but if you're developing for watches you're probably going to care about performance from the start and not have the GP's (completely valid in certain scenarios) attitude.

Re: High-performance .NET by example: Filtering bot traffic

#39

Post by alexandrnikitin » Mon, Apr 24, 2017, 7:32 PM UTC

deanCommie wrote: post #37

jackmott wrote: post #31

Earlier quoted context omitted.

Hardware might be cheap at some scales, but what if your service gets popular? What it latency for a single request can't be improved with more cores? What if your product is used by consumers who may have old hardware? or phones, or watches, or laptops and they want their battery to last? What if you consistently practiced at making high performance code, maybe then it wouldn't seem "unreadable" to you any more? Wha…

> what if your service gets popular? Then re-design for map-reduce, and scale horizontally > What it latency for a single request can't be improved with more cores? Then look into pre-computing and caching > What if your product is used by consumers who may have old hardware? or phones, or watches, or laptops and they want their battery to last? I thought we were talking about server requests? If we are, then offload…

> But this guy is writing server software. Micro-optimizing on the server side the way he is doing is silly.

Khm... What if you have millions of requests per second with tight latency requirements measured in milliseconds; and a bunch of business logic to fit into that. Such optimizations aren't so silly. There are different scenarios on both client and server sides.

Re: High-performance .NET by example: Filtering bot traffic

#40

Post by pc86 » Mon, Apr 24, 2017, 7:33 PM UTC

senorjazz wrote: post #8

Rather than block on UA, just add some honeypots. An invisible link. Any bot that pulls that page gets blocked as scrapers tend to pull all links from the page and follow. Use the robots.txt to ban the pulling of specific pages. Bots 99% of the time ignore robots, so if they pull it: block Check how quickly pages are pulled. If passes a threshold: block

I'm pretty sure the point of the article was performance testing of C#, not best practices for banning bots...