The QuestDB team are among the best doing it.
Love the people and their software.
Great blog Jaromir!
A 40-line fix eliminated a 400x performance gap
21–30 of 81 posts
Re: A 40-line fix eliminated a 400x performance gap
#22Earlier quoted context omitted.
Only for some clocks (CLOCK_MONOTONIC, etc) and some clock sources. For VIRT/SCHED, the vDSO shim still has to invoke the actual syscall. You can't avoid the kernel transition when you need per-thread accounting.
Thanks, I really should've looked deeper than that.
no problem at all, I was confused too when I saw the profile for the first time.
Re: A 40-line fix eliminated a 400x performance gap
#23> Flame graph image
> Click to zoom, open in a new tab for interactivity
I admit I did not expect "Open Image in New Tab" to do what it said on the tin. I guess I was aware that it was possible with SVG but I don't think I've ever seen it done and was really not expecting it.
Re: A 40-line fix eliminated a 400x performance gap
#24> Flame graph image > Click to zoom, open in a new tab for interactivity I admit I did not expect "Open Image in New Tab" to do what it said on the tin. I guess I was aware that it was possible with SVG but I don't think I've ever seen it done and was really not expecting it.
Courtesy of Brendan Gregg and his flamegraph.pl scripts: https://github.com/brendangregg/FlameGraph
Normally, I use the generator included in async-profiler. It produces interactive HTML. But for this post, I used Brendan’s tool specifically to have a single, interactive SVG.
Re: A 40-line fix eliminated a 400x performance gap
#25This is a great example of how a small change in the right place can outweigh years of incremental tuning.
Re: A 40-line fix eliminated a 400x performance gap
#26This is a great example of how a small change in the right place can outweigh years of incremental tuning.
I don't think I've ever seen less than 10x speedup after putting some effort into improving performance of "organic"/legacy code. It's always shocking how slow code can be before anyone complains.
Re: A 40-line fix eliminated a 400x performance gap
#27Flamegraphs are wonderful.
Me: looks at my code. "sure, ok, looks alright."
Me: looks at the resulting flamegraph. "what the hell is this?!?!?"
I've found all kinds of crazy stuff in codebases this way. Static initializers that aren't static, one-line logger calls that trigger expensive serialization, heavy string-parsing calls that don't memoize patterns, etc. Unfortunately some of those are my fault.
Re: A 40-line fix eliminated a 400x performance gap
#28It took seven years to address this concern following the initial bug report (2018). That seems like a lot, considering how instrumenting CPU time can be in the hot path for profiled code.
Re: A 40-line fix eliminated a 400x performance gap
#29Flamegraphs are wonderful. Me: looks at my code. "sure, ok, looks alright." Me: looks at the resulting flamegraph. "what the hell is this?!?!?" I've found all kinds of crazy stuff in codebases this way. Static initializers that aren't static, one-line logger calls that trigger expensive serialization, heavy string-parsing calls that don't memoize patterns, etc. Unfortunately some of those are my fault.
I also like icicle graphs for this. They're flamegraphs, but aggregated in the reverse order. (I.e. if you have calls A->B->C and D->E->C, then both calls to C are aggregated together, rather than being stacked on top of B and E respectively. It can make it easier to see what's wrong when you have a bunch of distinct codepaths that all invoke a common library where you're spending too much time.)
Regular flamegraphs are good too, icicle graphs are just another tool in the toolbox.
Re: A 40-line fix eliminated a 400x performance gap
#30Flamegraphs are wonderful. Me: looks at my code. "sure, ok, looks alright." Me: looks at the resulting flamegraph. "what the hell is this?!?!?" I've found all kinds of crazy stuff in codebases this way. Static initializers that aren't static, one-line logger calls that trigger expensive serialization, heavy string-parsing calls that don't memoize patterns, etc. Unfortunately some of those are my fault.
I might be very wrong in every way but, string parsing and or manipulating and memoiziation... sound like a super strange combo? For the first you know you're already doing expensive allocations, but the 2nd is also not a pattern I really see apart from in JS codebases. Could you provide more context on how this actually bit you in the behind? memoizing strings seems like a complicated and error prone "welp it feels better now" territory in my mind so I'm genuinely curious.