Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring Latency in Linux (2014)

btorpey.github.io

Post reply on HN
Print view View original

11–20 of 37 posts

Re: Measuring Latency in Linux (2014)

#11

Post by thedance »

cataphract wrote: post #10

I don't think his comment about CLOCK_MONOTONIC_RAW being slow to query applies anymore. It used to be slow because it was not implemented in the vDSO library and so it included the overhead of a syscall. But there was a big vDSO refactoring that landed on 5.3 that I think fixed this problem. Edit: found the patchset. In includes benchmarks for several architectures as well: https://lore.kernel.org/linux-arm-kernel/2…

It's a fun fact that on cloud VMs (AWS, etc) vDSO gettime doesn't exist, so if you rely on vDSO to make time measurement free, it's not.

Re: Measuring Latency in Linux (2014)

#12

Post by jstarks »

thedance wrote: post #11
cataphract wrote: post #10

I don't think his comment about CLOCK_MONOTONIC_RAW being slow to query applies anymore. It used to be slow because it was not implemented in the vDSO library and so it included the overhead of a syscall. But there was a big vDSO refactoring that landed on 5.3 that I think fixed this problem. Edit: found the patchset. In includes benchmarks for several architectures as well: https://lore.kernel.org/linux-arm-kernel/2…

It's a fun fact that on cloud VMs (AWS, etc) vDSO gettime doesn't exist, so if you rely on vDSO to make time measurement free, it's not.

Maybe this is true for AWS VMs that use Xen. I believe that Linux VMs on Azure do not have this problem, since they use the Hyper-V reference time page, which can be queried from the vDSO.

Re: Measuring Latency in Linux (2014)

#14

Post by coder543 »

vlovich123 wrote: post #8
krenel wrote: post #3

OP: Related stuff How Golang[1] implements monotonic clocks — basically it retrieves always both wall and monotonic clocks on "time.Now()" and when doing operations of subtraction it uses the monotonic. When printing, it uses the wall time. Pretty neat. Details on the proposal by Russ Cox [2]. [1] https://golang.org/pkg/time/#hdr-Monotonic_Clocks [2] https://go.googlesource.com/proposal/+/master/design/12914-m...

How does scheduling a timer to go off at 3pm this Saturday work?

That’s a hard question regardless of programming language. One commonly referenced YouTube video about time that explains why this is hard: https://youtu.be/-5wpm-gesOY

A reasonable first approximation of a solution would be to just check every second (or minute, or hour, depending on requirements) whether the current system time is later than the scheduled time for any pending events. Then you probably want to make sure events are marked as completed so they don’t fire again if the clock moves backwards.

Trying to predict how many seconds to sleep between now and 3pm on Saturday is a difficult task, but you can probably use a time library to do that if it’s important enough... but what happens when the government suddenly declares a sudden change to the time zone offset between now and then? The predictive solution would wake up at the wrong time.

Re: Measuring Latency in Linux (2014)

#15

Post by xakahnx »

thedance wrote: post #11
cataphract wrote: post #10

I don't think his comment about CLOCK_MONOTONIC_RAW being slow to query applies anymore. It used to be slow because it was not implemented in the vDSO library and so it included the overhead of a syscall. But there was a big vDSO refactoring that landed on 5.3 that I think fixed this problem. Edit: found the patchset. In includes benchmarks for several architectures as well: https://lore.kernel.org/linux-arm-kernel/2…

It's a fun fact that on cloud VMs (AWS, etc) vDSO gettime doesn't exist, so if you rely on vDSO to make time measurement free, it's not.

Any idea why this is? I'm even more curious since you've singled out "cloud VMs" from all VMs.

Re: Measuring Latency in Linux (2014)

#16

Post by matheusmoreira »

cataphract wrote: post #10

I don't think his comment about CLOCK_MONOTONIC_RAW being slow to query applies anymore. It used to be slow because it was not implemented in the vDSO library and so it included the overhead of a syscall. But there was a big vDSO refactoring that landed on 5.3 that I think fixed this problem. Edit: found the patchset. In includes benchmarks for several architectures as well: https://lore.kernel.org/linux-arm-kernel/2…

> The generic implementation includes the arch specific one and lives in "lib/vdso".

Is this the shared object that gets mapped to the address space of each process?

Re: Measuring Latency in Linux (2014)

#17

Post by thedance »

xakahnx wrote: post #15
thedance wrote: post #11

Earlier quoted context omitted.

It's a fun fact that on cloud VMs (AWS, etc) vDSO gettime doesn't exist, so if you rely on vDSO to make time measurement free, it's not.

Any idea why this is? I'm even more curious since you've singled out "cloud VMs" from all VMs.

My understanding is one of the reasons that the virtual time stuff in clouds doesn't work in a straightforward way is that your VM can migrate to another host, where reading TSC could give the appearance of discontinuous time, including jumping backwards in time which would be very bad. This is not a problem unless your VMs are migratory, which I guess is something I associate with GCE. And clouds seem to me to have more variety of hardware than I'd expect in my own private infrastructure, and that variety comes with a great many TSC quirks.

Re: Measuring Latency in Linux (2014)

#19

Post by hedora »

cataphract wrote: post #10

I don't think his comment about CLOCK_MONOTONIC_RAW being slow to query applies anymore. It used to be slow because it was not implemented in the vDSO library and so it included the overhead of a syscall. But there was a big vDSO refactoring that landed on 5.3 that I think fixed this problem. Edit: found the patchset. In includes benchmarks for several architectures as well: https://lore.kernel.org/linux-arm-kernel/2…

This is great news!

clock_monotonic greatly increases the failure surface of intra- (and inter-) machine timings than clock_monotonic_raw. A misconfigured ntp can cause bad slew in clock_monotonic. For clock_monotonic_raw, the main source of failures should be the oscillator controlling your CPU. If that happens, you have bigger problems.

Re: Measuring Latency in Linux (2014)

#20

Post by jsolson »

thedance wrote: post #17
xakahnx wrote: post #15

Earlier quoted context omitted.

Any idea why this is? I'm even more curious since you've singled out "cloud VMs" from all VMs.

My understanding is one of the reasons that the virtual time stuff in clouds doesn't work in a straightforward way is that your VM can migrate to another host, where reading TSC could give the appearance of discontinuous time, including jumping backwards in time which would be very bad. This is not a problem unless your VMs are migratory, which I guess is something I associate with GCE. And clouds seem to me to have…

Intel's VMCS includes a TSC offset field as well as TSC scaling. These allow for a stable RDTSC across migrations between hosts, modulo actual time lost to migration blackout.

(I work on virtualization in GCE)

Post reply on HN