Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 40-line fix eliminated a 400x performance gap

questdb.com

Post reply on HN
Print view View original

11–20 of 81 posts

Re: A 40-line fix eliminated a 400x performance gap

#11

Post by jacquesm »

rcxdude wrote: post #7
jacquesm wrote: post #6

Earlier quoted context omitted.

I don't think it is possible to talk about fractions of nanoseconds without having an extremely good idea of the stability and accuracy of your clock. At best I think you could claim there is some kind of reduction but it is super hard to make such claims in the absolute without doing a massive amount of prep work to ensure that the measured times themselves are indeed accurate. You could be off by a large fraction a…

Stability and accuracy, when applied to clocks, are generally about dynamic range, i.e. how good is the scale with which you are measuring time. So if you're talking about nanoseconds across a long time period, seconds or longer, then yeah, you probably should care about your clock. But when you're measuring nanoseconds out of a millisecond or microsecond, it really doesn't matter that much and you're going to be OK…

This setup is a user space program on a machine that is not exclusively dedicated to the test running all kinds of interrupts (and other tasks) left, right and center through the software under test.

Re: A 40-line fix eliminated a 400x performance gap

#13

Post by a-dub »

higherhalf wrote: post #12

clock_gettime() goes through vDSO, avoiding a context switch. It shows up on the flamegraph as well.

edit: agh, no. CLOCK_THREAD_CPUTIME_ID falls through the vdso to the kernel which makes sense as it would likely need to look at the task struct.

here it gets the task struct: https://elixir.bootlin.com/linux/v6.18.5/source/kernel/time/... and here https://elixir.bootlin.com/linux/v6.18.5/source/kernel/time/... to here where it actually pulls the value out: https://elixir.bootlin.com/linux/v6.18.5/source/kernel/sched...

where here is the vdso clock pick logic https://elixir.bootlin.com/linux/v6.18.5/source/lib/vdso/get... and here is the fallback to the syscall if it's not a vdso clock https://elixir.bootlin.com/linux/v6.18.5/source/lib/vdso/get...

Re: A 40-line fix eliminated a 400x performance gap

#14

Post by jerrinot »

higherhalf wrote: post #12

clock_gettime() goes through vDSO, avoiding a context switch. It shows up on the flamegraph as well.

Only for some clocks (CLOCK_MONOTONIC, etc) and some clock sources. For VIRT/SCHED, the vDSO shim still has to invoke the actual syscall. You can't avoid the kernel transition when you need per-thread accounting.

Re: A 40-line fix eliminated a 400x performance gap

#15

Post by ot »

higherhalf wrote: post #12

clock_gettime() goes through vDSO, avoiding a context switch. It shows up on the flamegraph as well.

If you look below the vDSO frame, there is still a syscall. I think that the vDSO implementation is missing a fast path for this particular clock id (it could be implemented though).

Re: A 40-line fix eliminated a 400x performance gap

#16

Post by jerrinot »

ot wrote: post #15
higherhalf wrote: post #12

clock_gettime() goes through vDSO, avoiding a context switch. It shows up on the flamegraph as well.

If you look below the vDSO frame, there is still a syscall. I think that the vDSO implementation is missing a fast path for this particular clock id (it could be implemented though).

Exactly this.

Re: A 40-line fix eliminated a 400x performance gap

#17

Post by ot »

You can do even faster, about 8ns (almost an additional 10x improvement) by using software perf events: PERF_COUNT_SW_TASK_CLOCK is thread CPU time, it can be read through a shared page (so no syscall, see perf_event_mmap_page), and then you add the delta since the last context switch with a single rdtsc call within a seqlock.

This is not well documented unfortunately, and I'm not aware of open-source implementations of this.

EDIT: Or maybe not, I'm not sure if PERF_COUNT_SW_TASK_CLOCK allows to select only user time. The kernel can definitely do it, but I don't know if the wiring is there. However this definitely works for overall thread CPU time.

Re: A 40-line fix eliminated a 400x performance gap

#18

Post by jerrinot »

ot wrote: post #17

You can do even faster, about 8ns (almost an additional 10x improvement) by using software perf events: PERF_COUNT_SW_TASK_CLOCK is thread CPU time, it can be read through a shared page (so no syscall, see perf_event_mmap_page), and then you add the delta since the last context switch with a single rdtsc call within a seqlock. This is not well documented unfortunately, and I'm not aware of open-source implementations…

That's a brilliant trick. The setup overhead and permission requirements for perf_event might be heavy for arbitrary threads, but for long-lived threads it looks pretty awesome! Thanks for sharing!

Re: A 40-line fix eliminated a 400x performance gap

#19

Post by ot »

jerrinot wrote: post #18
ot wrote: post #17

You can do even faster, about 8ns (almost an additional 10x improvement) by using software perf events: PERF_COUNT_SW_TASK_CLOCK is thread CPU time, it can be read through a shared page (so no syscall, see perf_event_mmap_page), and then you add the delta since the last context switch with a single rdtsc call within a seqlock. This is not well documented unfortunately, and I'm not aware of open-source implementations…

That's a brilliant trick. The setup overhead and permission requirements for perf_event might be heavy for arbitrary threads, but for long-lived threads it looks pretty awesome! Thanks for sharing!

Yes you need some lazy setup in thread-local state to use this. And short-lived threads should be avoided anyway :)

Re: A 40-line fix eliminated a 400x performance gap

#20

Post by higherhalf »

jerrinot wrote: post #14
higherhalf wrote: post #12

clock_gettime() goes through vDSO, avoiding a context switch. It shows up on the flamegraph as well.

Only for some clocks (CLOCK_MONOTONIC, etc) and some clock sources. For VIRT/SCHED, the vDSO shim still has to invoke the actual syscall. You can't avoid the kernel transition when you need per-thread accounting.

Thanks, I really should've looked deeper than that.
Post reply on HN