Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 40-line fix eliminated a 400x performance gap

questdb.com

Post reply on HN
Print view View original

61–70 of 81 posts

Re: A 40-line fix eliminated a 400x performance gap

#61

Post by jsymolon »

sroerick wrote: post #36
shermantanktop wrote: post #27

Flamegraphs are wonderful. Me: looks at my code. "sure, ok, looks alright." Me: looks at the resulting flamegraph. "what the hell is this?!?!?" I've found all kinds of crazy stuff in codebases this way. Static initializers that aren't static, one-line logger calls that trigger expensive serialization, heavy string-parsing calls that don't memoize patterns, etc. Unfortunately some of those are my fault.

I've never used flamegraphs but would like to know about them. Can you explain more? Or where should I start?

I use them all the time on Perl code.

https://metacpan.org/pod/Devel::NYTProf

Re: A 40-line fix eliminated a 400x performance gap

#62

Post by arethuza »

shermantanktop wrote: post #27

Flamegraphs are wonderful. Me: looks at my code. "sure, ok, looks alright." Me: looks at the resulting flamegraph. "what the hell is this?!?!?" I've found all kinds of crazy stuff in codebases this way. Static initializers that aren't static, one-line logger calls that trigger expensive serialization, heavy string-parsing calls that don't memoize patterns, etc. Unfortunately some of those are my fault.

I always found profiling performance critical code and experimenting with optimisations to be one of the most enjoyable parts of development - probably because of the number of surprises that I encountered ("Why on Earth is that so slow?").

Re: A 40-line fix eliminated a 400x performance gap

#63

Post by jerrinot »

jacquesm wrote: post #60
loeg wrote: post #41

Earlier quoted context omitted.

This seems like more of a philosophical argument than a practical one.

No, it is a very practical one and I'm actually surprised that you don't see it that way. Benchmarking is hard, and if you don't understand the basics then you can easily measure nonsense.

You raise a fair point about the percentiles. Those are reported as point estimates without confidence intervals and the implied precision overstates what system clock can deliver.

The mean does get proper statistical treatment (t-distribution confidence interval), but you're right that JMH doesn't compute confidence intervals for percentiles. Reporting p0.00 with three significant figures is ... optimistic.

That said I think the core finding survives this critique. The improvement shows up consistently across ~11 million samples at every percentile from p0.50 through p0.999.

Re: A 40-line fix eliminated a 400x performance gap

#64

Post by burnt-resistor »

I really wished™ there was an API/ABI for userland- and kernelland-defined individual virtual files at arbitrary locations, backed by processes and kernel modules respectively. I've tried pipes, overlays, and FUSE to no avail. It would greatly simply configuration management implementations while maintaining compatibility with the convention of plain text files, and there's often no need to have an actual file on any media or the expense of IOPS.

While I don't particularly like the IO overhead and churn consequences of real files for performance metrics, I get the 9p-like appeal of treating the virtual fs as a DBMS/API/ABI.

Re: A 40-line fix eliminated a 400x performance gap

#65

Post by u8080 »

loeg wrote: post #34
otterley wrote: post #28

It took seven years to address this concern following the initial bug report (2018). That seems like a lot, considering how instrumenting CPU time can be in the hot path for profiled code.

400x slower than 70ns is still only 28us. How often is the JVM calling this function?

28us is still solid amount of time

Re: A 40-line fix eliminated a 400x performance gap

#67

Post by mgaunard »

ot wrote: post #17

You can do even faster, about 8ns (almost an additional 10x improvement) by using software perf events: PERF_COUNT_SW_TASK_CLOCK is thread CPU time, it can be read through a shared page (so no syscall, see perf_event_mmap_page), and then you add the delta since the last context switch with a single rdtsc call within a seqlock. This is not well documented unfortunately, and I'm not aware of open-source implementations…

clock_gettime is not doing a syscall, it's using vdso.

Re: A 40-line fix eliminated a 400x performance gap

#68

Post by jerrinot »

mgaunard wrote: post #67
ot wrote: post #17

You can do even faster, about 8ns (almost an additional 10x improvement) by using software perf events: PERF_COUNT_SW_TASK_CLOCK is thread CPU time, it can be read through a shared page (so no syscall, see perf_event_mmap_page), and then you add the delta since the last context switch with a single rdtsc call within a seqlock. This is not well documented unfortunately, and I'm not aware of open-source implementations…

clock_gettime is not doing a syscall, it's using vdso.

clock_gettime() goes through the vDSO shim, but whether it avoids a syscall depends on the clock ID and (in some cases) the clock source. For thread-specific CPU user time, the vDSO shim cannot resolve the request in user space and must transit into the kernel. In this specific case, there is absolutely a syscall.

Re: A 40-line fix eliminated a 400x performance gap

#69

Post by jacquesm »

jerrinot wrote: post #63
jacquesm wrote: post #60

Earlier quoted context omitted.

No, it is a very practical one and I'm actually surprised that you don't see it that way. Benchmarking is hard, and if you don't understand the basics then you can easily measure nonsense.

You raise a fair point about the percentiles. Those are reported as point estimates without confidence intervals and the implied precision overstates what system clock can deliver. The mean does get proper statistical treatment (t-distribution confidence interval), but you're right that JMH doesn't compute confidence intervals for percentiles. Reporting p0.00 with three significant figures is ... optimistic. That sai…

Yes, I would expect the 'order of magnitude' value to be relatively close but the absolute values to be very imprecise.

Re: A 40-line fix eliminated a 400x performance gap

#70

Post by singron »

otterley wrote: post #44
loeg wrote: post #34

Earlier quoted context omitted.

400x slower than 70ns is still only 28us. How often is the JVM calling this function?

It depends. If you’re doing continuous profiling, it’d make a call to get the current time at every method entry and exit, each of which could then add a context switch. In an absolute sense it appears to be small, but it could really add up. This is what flame graphs are super helpful for, to see whether it’s really a problem or not. Also, remember that every extra moment running instructions is a lost opportunity t…

If you are doing continuous profiling, you are probably using a low overhead stack sampling profiler rather than recording every method entry and exit.
Post reply on HN