Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High-performance .NET by example: Filtering bot traffic

alexandrnikitin.github.io

Post reply on HN
Print view View original

41–50 of 54 posts

Re: High-performance .NET by example: Filtering bot traffic

#41

Post by hdhzy »

Someone1234 wrote: post #19
hdhzy wrote: post #13

Excellent post showing how to correctly improve code w.r.t. performance using the scientific method: hypothesis, measuring the baseline, change, measuring effect with real tools, real code. Thanks for sharing!

I also found it an interesting post in that it kind of inadvertently proves that for most situations you shouldn't optimise to this extent. Meaning, yes, the OP got impressive performance improvements but the code is also completely unreadable and utilises unsafe code sections which could expose you to security problems/memory leaks/memory corruption. Not to mention they've recreated and will need to maintain an in-h…

Well I don't think the author advises to go all the way down with your code but rather shows how to do that if you want to. As usual - what will anyone do with that knowledge is up to them but you surely will not advise not using a tool because you can do bad things with it.

Re: High-performance .NET by example: Filtering bot traffic

#42

Post by hdhzy »

Someone1234 wrote: post #35
osd wrote: post #25

Earlier quoted context omitted.

No, no no no. I cannot believe that you are honestly saying a 2x increase to throughput in production is something you "shouldn't take seriously" because the code isn't as readable as it was before. Programmers are expensive. Hardware is cheap. That doesn't justify completely throwing out the window any performance increasing changes just because a fresh college grad won't be able to understand what's going on within…

> I cannot believe that you are honestly saying a 2x increase to throughput in production is something you "shouldn't take seriously" because the code isn't as readable as it was before. I've worked on large and old codebases for years. I've seen plenty of examples of where a "clever" programmer has optimised the heck out of a section of code, made it completely unmaintainable, and as a result forced a re-write (the…

I don't know your experiences but the "performance optimizations" that I've seen lacked obvious technical support around it. Usually people thought "it will be faster" without using profiler (sin no 1) or did not properly isolate and control the performance critical part of the code (making performance tests part of the build etc.). But these are other issues separate from optimizing code.

The usual caveat of making code correct, readable and then fast in this order of course also applies :)

Re: High-performance .NET by example: Filtering bot traffic

#43

Post by osd »

Someone1234 wrote: post #35
osd wrote: post #25

Earlier quoted context omitted.

No, no no no. I cannot believe that you are honestly saying a 2x increase to throughput in production is something you "shouldn't take seriously" because the code isn't as readable as it was before. Programmers are expensive. Hardware is cheap. That doesn't justify completely throwing out the window any performance increasing changes just because a fresh college grad won't be able to understand what's going on within…

> I cannot believe that you are honestly saying a 2x increase to throughput in production is something you "shouldn't take seriously" because the code isn't as readable as it was before. I've worked on large and old codebases for years. I've seen plenty of examples of where a "clever" programmer has optimised the heck out of a section of code, made it completely unmaintainable, and as a result forced a re-write (the…

I have worked on large codebases. You sound less like you have a solid opinion backed up by experience and more like you enjoy spouting platitudes and best practices in meetings without getting dirty with too much code yourself.

Re: High-performance .NET by example: Filtering bot traffic

#44

Post by Someone1234 »

osd wrote: post #43
Someone1234 wrote: post #35

Earlier quoted context omitted.

> I cannot believe that you are honestly saying a 2x increase to throughput in production is something you "shouldn't take seriously" because the code isn't as readable as it was before. I've worked on large and old codebases for years. I've seen plenty of examples of where a "clever" programmer has optimised the heck out of a section of code, made it completely unmaintainable, and as a result forced a re-write (the…

I have worked on large codebases. You sound less like you have a solid opinion backed up by experience and more like you enjoy spouting platitudes and best practices in meetings without getting dirty with too much code yourself.

That sounds more like a personal attack than a constructive response to the topic.

I only brought up large (old) codebases to put into context the kind of maintainability issues I am discussing, and why my perspective is what it is.

Re: High-performance .NET by example: Filtering bot traffic

#45

Post by mlvljr »

osd wrote: post #43
Someone1234 wrote: post #35

Earlier quoted context omitted.

> I cannot believe that you are honestly saying a 2x increase to throughput in production is something you "shouldn't take seriously" because the code isn't as readable as it was before. I've worked on large and old codebases for years. I've seen plenty of examples of where a "clever" programmer has optimised the heck out of a section of code, made it completely unmaintainable, and as a result forced a re-write (the…

I have worked on large codebases. You sound less like you have a solid opinion backed up by experience and more like you enjoy spouting platitudes and best practices in meetings without getting dirty with too much code yourself.

how insightful :V

Re: High-performance .NET by example: Filtering bot traffic

#46

Post by Twirrim »

throwasehasdwi wrote: post #2

Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so you're only blocking well behaved ones. Plus there's thousands of browser versions out there, so there's a very good chance you're blocking some users for no reason. The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the si…

The article is not about filtering bad bots. It even says so right near the top:

>We won’t cover black bots because it is a huge topic with sophisticated analysis and Machine learning algorithms. We will focus on the white and grey bots that identify themselves as such.

This is about not wasting time & effort showing advertising banners to good bots.

Re: High-performance .NET by example: Filtering bot traffic

#47

Post by doubleplusgood »

oblio wrote: post #33
doubleplusgood wrote: post #14

I did something similar with nginx, the data file from 51degrees and some lua code; each instance only handles 10-20k requests/sec so no clever optimization was needed.

Would you mind posting the Lua code?

Sure, but there's nothing really special, just some json parsing and some variable assignments back into an internal nginx request. I'm on mobile at the moment but I'll follow up on this in the next 24 hours.

Re: High-performance .NET by example: Filtering bot traffic

#48

Post by osd »

Someone1234 wrote: post #44
osd wrote: post #43

Earlier quoted context omitted.

I have worked on large codebases. You sound less like you have a solid opinion backed up by experience and more like you enjoy spouting platitudes and best practices in meetings without getting dirty with too much code yourself.

That sounds more like a personal attack than a constructive response to the topic. I only brought up large (old) codebases to put into context the kind of maintainability issues I am discussing, and why my perspective is what it is.

It's not a personal attack because I don't actually know anything about you, and I'm not claiming to. I'm making that statement in regard to your two posts up the chain. They are statements that don't reveal any solid, usable points other than to say "simpler code is easier to understand, sometimes performance doesn't matter as much" but in such a strong form as to suggest that the advice plays out a lot in the real world. It doesn't.

Re: High-performance .NET by example: Filtering bot traffic

#49

Post by dahauns »

alexandrnikitin wrote: post #39
deanCommie wrote: post #37

Earlier quoted context omitted.

> what if your service gets popular? Then re-design for map-reduce, and scale horizontally > What it latency for a single request can't be improved with more cores? Then look into pre-computing and caching > What if your product is used by consumers who may have old hardware? or phones, or watches, or laptops and they want their battery to last? I thought we were talking about server requests? If we are, then offload…

> But this guy is writing server software. Micro-optimizing on the server side the way he is doing is silly. Khm... What if you have millions of requests per second with tight latency requirements measured in milliseconds; and a bunch of business logic to fit into that. Such optimizations aren't so silly. There are different scenarios on both client and server sides.

Heh. You mean, like the example in the article? :)

Re: High-performance .NET by example: Filtering bot traffic

#50

Post by ro_sharp »

deanCommie wrote: post #37
jackmott wrote: post #31

Earlier quoted context omitted.

Hardware might be cheap at some scales, but what if your service gets popular? What it latency for a single request can't be improved with more cores? What if your product is used by consumers who may have old hardware? or phones, or watches, or laptops and they want their battery to last? What if you consistently practiced at making high performance code, maybe then it wouldn't seem "unreadable" to you any more? Wha…

> what if your service gets popular? Then re-design for map-reduce, and scale horizontally > What it latency for a single request can't be improved with more cores? Then look into pre-computing and caching > What if your product is used by consumers who may have old hardware? or phones, or watches, or laptops and they want their battery to last? I thought we were talking about server requests? If we are, then offload…

I'd suggest that OLTP applications where non-trivial amounts of state are shared between transactions are a reasonable counterexample where single core performance optimisation really matters.
Post reply on HN