I wonder why the blog post claims setting clock source to 'tsc' is considered dangerous.
Because if the clock rate changes, tsc can become out of sync.
Two frequently used system calls are ~77% slower on AWS EC2
41–50 of 101 posts
Re: Two frequently used system calls are ~77% slower on AWS EC2
#42Interesting, I ran into this issue in PHP a while back. I found a really basic loop that ran thousands of times per pageload that was running REALLY slowly. Caching the time before running the loop solved the problem entirely.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#43Yes, this is why we (Netflix) default to tsc over the xen clocksource. I found the xen clocksource had become a problem a few years ago, quantified using flame graphs, and investigated using my own microbenchmark. Summarized details here: https://www.slideshare.net/brendangregg/performance-tuning-e...
Honestly everyone should be defaulting to the TSC on modern x86. Timekeeping on a single OS image over the short term[1] is a hardware feature available at the ISA level now. It's not something to which the OS can add value, and as we see in circumstances like this it tends to muck things up trying to abstract it.
[1] Long term issues like inter-clock drift and global synchronization are a rather different problem area, and the OS has tools to help there.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#44The title is misleading. 77% slower sounds like the system calls take 1.77x the time on EC2. In fact, the results indicate that the normal calls are 77% faster - in other words, EC2 gettimeofday and clock_gettime calls take nearly 4.5x longer to run on EC2 than they do on ordinary systems. This is a big speed hit. Some programs can use gettimeofday extremely frequently - for example, many programs call timing functio…
77% faster is not correct either. "Speed" would probably by ops/s.
4.5x longer = 350% slower.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#45I wonder why the blog post claims setting clock source to 'tsc' is considered dangerous.
Because if the clock rate changes, tsc can become out of sync. https://lwn.net/Articles/209101/
Not really. Recent CPUs (at least those from Intel, which is what EC2 runs on) implement constant_tsc, so the frequency does not affect the tsc.
A worse issue is that the counters may not be synchronized between cpus, which may be an issue when the process moves between sockets.
But I wouldn't call that "dangerous", it's simply a feature of the clock source. If that's an issue for your program, you should use CLOCK_MONOTONIC anyway and not rely on gettimeofday() doing the right thing.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#46I prefer the way Solaris solved this problem: 1) first, by eliminating the need for a context switch for libc calls such as gettimeofday(), gethrtime(), etc. (there is no public/supported interface on Solaris for syscalls, so libc would be used) 2) by providing additional, specific interfaces with certain guarantees: https://docs.oracle.com/cd/E53394_01/html/E54766/get-sec-fro... This was accomplished by creating a s…
This is precisely what the vDSO does. The clocksources mentioned explicitly list themselves as not supporting this action, hence the fallback to a regular system call.
Not quite; vdso is a general syscall-wrapper mechanism. The Solaris solution is specifically just for the gettimeofday(), gethrtime() interfaces, etc.
The difference is that on Solaris, since there is no public system call interface, there's also no need for a fallback. Every program is just faster, no matter how Solaris is virtualized, since every program is using libc.
There's also no need for an administrative interface to control clocksource; the best one is always used.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#47I prefer the way Solaris solved this problem: 1) first, by eliminating the need for a context switch for libc calls such as gettimeofday(), gethrtime(), etc. (there is no public/supported interface on Solaris for syscalls, so libc would be used) 2) by providing additional, specific interfaces with certain guarantees: https://docs.oracle.com/cd/E53394_01/html/E54766/get-sec-fro... This was accomplished by creating a s…
You just described the old method Linux used that was vulnerable to info leaks iirc and why it now a vDSO
The Solaris method doesn't have the problem the other implementation did.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#48Author here, greetings. Anyone who finds this interesting may also enjoy our writeup describing every Linux system call method in detail [1]. [1]: https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g...
Nitpick: slower _than_ what? It's implied, but "slower" (or "greater", or anything-er) is in relation to another thing.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#49Is this just an EC2 problem, or does it affect any Xen/KVM guest? I ran the test program on a Hyper-V VM running CentOS 7 and got the same result: 100 calls to the gettimeofday syscall. Conversely, I tested a vSphere guest (also running CentOS 7), which didn't call gettimeofday at all.
>Is this just an EC2 problem, or does it affect any Xen/KVM guest? Looks like it's how the Xen hypervisor works.
It is slower because it misses an optimization where you can get the current time without having to enter the kernel. The trick is using the RDTSC instruction, which is not a privileged instruction, so you can call it from userspace. The Time Stamp Counter is a 64 bit register (MSR actually), which gets incremented monotonically. You can get the current time by calibrating it against a known duration on boot or get the frequency from a system table first, then with a simple division and adding an offset. There are sone caveats though, like you have to check if the CPU has an invariant TSC using CPUID and every core has a separate register. I think the problem with XEN is that the VM could be moved across hypervisors or CPUs which would suddenly change the value of the counter. The latter could be mitigated by syncing the TSCs across cores (did I mention that they are writable?) and XEN supports emulating the RDTSC instruction too. I'm not sure how it's configured on AWS, so it may be perfectly safe or mostly safe.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#50Earlier quoted context omitted.
Because if the clock rate changes, tsc can become out of sync. https://lwn.net/Articles/209101/
Not really. Recent CPUs (at least those from Intel, which is what EC2 runs on) implement constant_tsc, so the frequency does not affect the tsc. A worse issue is that the counters may not be synchronized between cpus, which may be an issue when the process moves between sockets. But I wouldn't call that "dangerous", it's simply a feature of the clock source. If that's an issue for your program, you should use CLOCK_M…
how does constant_tsc interact with VMs being silently migrated from one physical machine to another?