Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring Latency in Linux (2014)

btorpey.github.io

Post reply on HN
Print view View original

21–30 of 37 posts

Re: Measuring Latency in Linux (2014)

#21

Post by BeeOnRope »

The thing about needing cpuid isnt true except perhaps on some older AMD hardware.

lfence works as a execution barrier and has an explicit cost of only a few cycles. You can accurately time a region with something like:

    lfence
    rdtsc
    lfence
    // timed region
    lfence
    rdtsc
This will give you accurate timing with some offset (i.e. even with an empty region you get a result on the order of 25-40 cycles), which you can mostly subtract out.

Carefully done you can get results down to a nanosecond or so.

rdtscp has few advantages over lfence + rdtsc, and arguably some disadvantages (you can control where the implied fence goes).

Re: Measuring Latency in Linux (2014)

#22

Post by ggm »

How does VM affect this?

How does KVM affect this?

How does Docker on KVM affect this?

How does Hypervisor affect this?

Add "... for a given network driver, e2e, measured RTT.."

Re: Measuring Latency in Linux (2014)

#23

Post by BeeOnRope »

jsolson wrote: post #20
thedance wrote: post #17

Earlier quoted context omitted.

My understanding is one of the reasons that the virtual time stuff in clouds doesn't work in a straightforward way is that your VM can migrate to another host, where reading TSC could give the appearance of discontinuous time, including jumping backwards in time which would be very bad. This is not a problem unless your VMs are migratory, which I guess is something I associate with GCE. And clouds seem to me to have…

Intel's VMCS includes a TSC offset field as well as TSC scaling. These allow for a stable RDTSC across migrations between hosts, modulo actual time lost to migration blackout. (I work on virtualization in GCE)

So do gettimeofday and the various clock_gettime methods [1] hit the vDSO on GCP, or do they incur a syscall, or something else?

---

[1] Not all of the clock_gettime sources hit the vDSO even on bare metal Linux on typical x86 hardware, but many of the important ones do.

Re: Measuring Latency in Linux (2014)

#24

Post by jasonzemos »

BeeOnRope wrote: post #21

The thing about needing cpuid isnt true except perhaps on some older AMD hardware. lfence works as a execution barrier and has an explicit cost of only a few cycles. You can accurately time a region with something like: lfence rdtsc lfence // timed region lfence rdtsc This will give you accurate timing with some offset (i.e. even with an empty region you get a result on the order of 25-40 cycles), which you can mostl…

Specifically, the Intel manual makes the following important points, one involving an `mfence;lfence` combo:

* If software requires RDTSC to be executed only after all previous instructions have executed and all previous loads are globally visible, it can execute LFENCE immediately before RDTSC.

* If software requires RDTSC to be executed only after all previous instructions have executed and all previous loads and stores are globally visible, it can execute the sequence MFENCE;LFENCE immediately before RDTSC.

* If software requires RDTSC to be executed prior to execution of any subsequent instruction (including any memory accesses), it can execute the sequence LFENCE immediately after RDTSC. This instruction was introduced by the Pentium processor.

rdtscp is usually a bit more disruptive, and cpuid is probably 100 or 1000 times more disruptive.

Re: Measuring Latency in Linux (2014)

#25

Post by saagarjha »

matheusmoreira wrote: post #16
cataphract wrote: post #10

I don't think his comment about CLOCK_MONOTONIC_RAW being slow to query applies anymore. It used to be slow because it was not implemented in the vDSO library and so it included the overhead of a syscall. But there was a big vDSO refactoring that landed on 5.3 that I think fixed this problem. Edit: found the patchset. In includes benchmarks for several architectures as well: https://lore.kernel.org/linux-arm-kernel/2…

> The generic implementation includes the arch specific one and lives in "lib/vdso". Is this the shared object that gets mapped to the address space of each process?

Yes.

Re: Measuring Latency in Linux (2014)

#26

Post by saagarjha »

thedance wrote: post #17
xakahnx wrote: post #15

Earlier quoted context omitted.

Any idea why this is? I'm even more curious since you've singled out "cloud VMs" from all VMs.

My understanding is one of the reasons that the virtual time stuff in clouds doesn't work in a straightforward way is that your VM can migrate to another host, where reading TSC could give the appearance of discontinuous time, including jumping backwards in time which would be very bad. This is not a problem unless your VMs are migratory, which I guess is something I associate with GCE. And clouds seem to me to have…

Wouldn't preventing access to high-resolution clocks also be a mitigation for speculative side channels?

Re: Measuring Latency in Linux (2014)

#28

Post by xakahnx »

jsolson wrote: post #20
thedance wrote: post #17

Earlier quoted context omitted.

My understanding is one of the reasons that the virtual time stuff in clouds doesn't work in a straightforward way is that your VM can migrate to another host, where reading TSC could give the appearance of discontinuous time, including jumping backwards in time which would be very bad. This is not a problem unless your VMs are migratory, which I guess is something I associate with GCE. And clouds seem to me to have…

Intel's VMCS includes a TSC offset field as well as TSC scaling. These allow for a stable RDTSC across migrations between hosts, modulo actual time lost to migration blackout. (I work on virtualization in GCE)

Cool! I looked up more info on this and ended up here, http://www.brendangregg.com/blog/2017-05-04/the-pmcs-of-ec2....

But I only skimmed this and I don't see that it mentions about migration across hosts. Only that the hypervisor is able to expose the MSRs or PMCs.

Re: Measuring Latency in Linux (2014)

#29

Post by robbjuly »

jstarks wrote: post #12
thedance wrote: post #11

Earlier quoted context omitted.

It's a fun fact that on cloud VMs (AWS, etc) vDSO gettime doesn't exist, so if you rely on vDSO to make time measurement free, it's not.

Maybe this is true for AWS VMs that use Xen. I believe that Linux VMs on Azure do not have this problem, since they use the Hyper-V reference time page, which can be queried from the vDSO.

I believe newer generation AWS VMs, like C5, use kvm clock source now and not xen. On the older ones switching to tsc speeds up things.
Post reply on HN