How common are get time calls so that they would actually be an issue?
I've worked on quite a few systems and can't think of a time where an api for getting the time would have been called so much that it would affect performance?
Two frequently used system calls are ~77% slower on AWS EC2
31–40 of 101 posts
Re: Two frequently used system calls are ~77% slower on AWS EC2
#32Yes, this is why we (Netflix) default to tsc over the xen clocksource. I found the xen clocksource had become a problem a few years ago, quantified using flame graphs, and investigated using my own microbenchmark. Summarized details here: https://www.slideshare.net/brendangregg/performance-tuning-e...
Can you share if you needed to do anything to deal with time drift issues when using tsc? For my own systems, incorrect timestamps would cause a lot of issues.
Well, it's been a few years and we haven't switched it back. :)
We have had a number of clock issues, and one of the first things I try is taking and instance and switching it back to xen for a few days, but those issues have not turned out to be the clocksource. Usually NTP.
AWS can comment more about the state (safety/risk) of these clocksources (given they have access to all the SW/HW internals).
Re: Two frequently used system calls are ~77% slower on AWS EC2
#33Yes, this is why we (Netflix) default to tsc over the xen clocksource. I found the xen clocksource had become a problem a few years ago, quantified using flame graphs, and investigated using my own microbenchmark. Summarized details here: https://www.slideshare.net/brendangregg/performance-tuning-e...
This reminds me: I should give an updated version of that talk for 2017...
Re: Two frequently used system calls are ~77% slower on AWS EC2
#34Interesting way to find out the version of the hypervisor kernel. If the gtod call returns faster than the direct syscall for it, then you know the kernel version is prior to that of the patch fixing the issue in xen.
I expect there are many such patches that you could use to narrow down the version range of the host kernel. Once you've that information, you may be in a better position to exploit it, knowing which bugs are and are not patched.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#35I wonder why the blog post claims setting clock source to 'tsc' is considered dangerous.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#36The title is misleading. 77% slower sounds like the system calls take 1.77x the time on EC2. In fact, the results indicate that the normal calls are 77% faster - in other words, EC2 gettimeofday and clock_gettime calls take nearly 4.5x longer to run on EC2 than they do on ordinary systems.
This is a big speed hit. Some programs can use gettimeofday extremely frequently - for example, many programs call timing functions when logging, performing sleeps, or even constantly during computations (e.g. to implement a poor-man's computation timeout).
The article suggests changing the time source to tsc as a workaround, but also warns that it could cause unwanted backwards time warps - making it dangerous to use in production. I'd be curious to hear from those who are using it in production how they avoided the "time warp" issue.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#37Another option is to reduce usage of gettimeofday() when possible. It is not always free. Roughly 10 years ago, when I was the driver author for one of the first full-speed 10GbE NICs, we'd get complaints from customers that were sure our NIC could not do 10Gbs, as iperf showed it was limited to 3Gb/s or less. I would ask them to re-try with netperf, and they'd see full bandwidth. I eventually figured out that the co…
> Another option is to reduce usage of gettimeofday() when possible. It is not always free.
haha, it's amazing how much software is written that basically does something ridiculous like "while (gettimeofday()) clock_gettime();".
Re: Two frequently used system calls are ~77% slower on AWS EC2
#38How common are get time calls so that they would actually be an issue? I've worked on quite a few systems and can't think of a time where an api for getting the time would have been called so much that it would affect performance?
Timestamped logs, transaction timeouts, http keepalive timeouts, cache expiration/eviction, etc.
Apache and nginx for example, both call gettimeofday() a lot.
Edit: Quick google searches indicate software like redis and memcached also call it quite often.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#39OpenJDK has an open issue about this in their JVM: https://bugs.openjdk.java.net/browse/JDK-8165437
Re: Two frequently used system calls are ~77% slower on AWS EC2
#40So… it's not that the syscalls are slower, it's that the Linux-specific mechanism the Linux kernel uses to bypass having to actually perform these calls does not currently work on Xen (and thus EC2).
Depends on if you're looking at this from userspace or kernelspace. From the latter, you're spot on. From the former, the headline's spot on.
> From the former, the headline's spot on.
Only if you're using Linux guests and assuming vDSO so not really. The headline made me first go to issues with the host/virtual hardware and some syscalls being much slower than normal across the board.