Two frequently used system calls are ~77% slower on AWS EC2 • Hacker News Board

Re: Two frequently used system calls are ~77% slower on AWS EC2

#61

Post by poofyleek » Tue, Mar 07, 2017, 11:54 PM UTC

pgaddict wrote: post #59

poofyleek wrote: post #57

Earlier quoted context omitted.

Even without migration, the synchronization can be an issue. In older multi-core machines, tsc synchronization was an issue among cores. Modern systems take care of this. And core CPU clock frequency change is also taken care of, so that constant rate is available via tsc. However, when hypervisors such as VMWare or paravirtualization like Xen come into play, there are further issues, because RDTSC instruction either…

It's complex stuff, no doubt about that. For me, it's much simpler - I come from the PostgreSQL world, so gettimeofday() is pretty much what EXPLAIN ANALYZE does to instrument queries. Good time source means small overhead, bad time source means instrumented queries may take multiples of actual run time (and be skewed in various ways). No fun.

It is complex and interesting. I am a novice database user. But I do know many databases use 'gettimeofday' quite a lot. Just strace any SELECT query. Most databases I have used, including Postgresql, also have to implement MVCC which mostly depend on timestamps. Imagine the hypervisor CPU and memory pressure induced time drift, or even drift in distributed cluster of database nodes. It hurts my head to think of the cases that will give me the wrong values or wrong estimate for getting the values. It is an interesting area.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#62

Post by [deleted] » Tue, Mar 07, 2017, 11:54 PM UTC

pgaddict wrote: post #59

poofyleek wrote: post #57

Earlier quoted context omitted.

Even without migration, the synchronization can be an issue. In older multi-core machines, tsc synchronization was an issue among cores. Modern systems take care of this. And core CPU clock frequency change is also taken care of, so that constant rate is available via tsc. However, when hypervisors such as VMWare or paravirtualization like Xen come into play, there are further issues, because RDTSC instruction either…

It's complex stuff, no doubt about that. For me, it's much simpler - I come from the PostgreSQL world, so gettimeofday() is pretty much what EXPLAIN ANALYZE does to instrument queries. Good time source means small overhead, bad time source means instrumented queries may take multiples of actual run time (and be skewed in various ways). No fun.

[deleted]

Re: Two frequently used system calls are ~77% slower on AWS EC2

#63

Post by Anderkent » Tue, Mar 07, 2017, 11:56 PM UTC

tyingq wrote: post #38

teddyuk wrote: post #31

How common are get time calls so that they would actually be an issue? I've worked on quite a few systems and can't think of a time where an api for getting the time would have been called so much that it would affect performance?

Timestamped logs, transaction timeouts, http keepalive timeouts, cache expiration/eviction, etc. Apache and nginx for example, both call gettimeofday() a lot. Edit: Quick google searches indicate software like redis and memcached also call it quite often.

So does cassandra.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#64

Post by codyps » Tue, Mar 07, 2017, 11:56 PM UTC

binarycrusader wrote: post #11

I prefer the way Solaris solved this problem: 1) first, by eliminating the need for a context switch for libc calls such as gettimeofday(), gethrtime(), etc. (there is no public/supported interface on Solaris for syscalls, so libc would be used) 2) by providing additional, specific interfaces with certain guarantees: https://docs.oracle.com/cd/E53394_01/html/E54766/get-sec-fro... This was accomplished by creating a s…

For those actually curious about the implementation on solaris/illumos, heres a quick rundown (from looking at current illumos source):

- comm_page (usr/src/uts/i86pc/ml/comm_page.s) is literally a page in kernel memory with specific variables that is mapped (usr/src/uts/intel/ia32/os/comm_page_util.c) as user|read-only (to be passed to userspace, kernel mapping is normal data, AFAICT)

- the mapped comm_page is inserted into the aux vector at AT_SUN_COMMPAGE (usr/src/uts/common/exec/elf/elf.c)

- libc scans auxv for this entry, and stashes the pointer it containts (usr/src/lib/libc/port/threads/thr.c)

- When clock_gettime is called, it looks at the values in the COMMPAGE (structure is in usr/src/uts/i86pc/sys/comm_page.h, probing in usr/src/lib/commpage/common/cp_main.c) to determine if TSC can be used.

- If TSC is usable, libc uses the information there (a bunch of values) to use tsc to read time (monotonic or realtime)

Variables within comm_page are treated like normal variables and used/updated within the kernel's internal timekeeping.

Essentially, rather than having the kernel provide an entry point & have the kernel know what the (in the linux case) internal data structures look like, here libc provides the code and reads the exported data structure from the kernel.

So it isn't reading the time from this memory page, it's using TSC. In the case of CLOCK_REALTIME, corrections that are applied to TSC are read from this memory page (comm_page).

Re: Two frequently used system calls are ~77% slower on AWS EC2

#65

Post by wahern » Wed, Mar 08, 2017, 12:06 AM UTC

ajross wrote: post #43

brendangregg wrote: post #10

Yes, this is why we (Netflix) default to tsc over the xen clocksource. I found the xen clocksource had become a problem a few years ago, quantified using flame graphs, and investigated using my own microbenchmark. Summarized details here: https://www.slideshare.net/brendangregg/performance-tuning-e...

Honestly everyone should be defaulting to the TSC on modern x86. Timekeeping on a single OS image over the short term[1] is a hardware feature available at the ISA level now. It's not something to which the OS can add value, and as we see in circumstances like this it tends to muck things up trying to abstract it. [1] Long term issues like inter-clock drift and global synchronization are a rather different problem ar…

The gettimeofday vDSO does use the TSC. The purpose of the vDSO is making visible the continuously updated values necessary for userland to adjust and correct TSC-based calculations. Many of those values are still necessary even when the TSC is shared and constant-rate.

A pure TSC implementation will sacrifice accuracy (because it's not being trained by the HPET or corrected by NTP), performance (because it'll need to do a full syscall occasionally), or both.

If you're sophisticated like NetFlix you can probably assure yourself it's no big deal. But it's a bad idea for others to blindly do the same thing. Look at the issue with Go's timeouts. Go used gettimeofday rather than CLOCK_MONOTONIC because the authors assumed the behavior of Google's system's clock skewing algorithm. That assumption broke spectacularly for many other people not using Google's servers.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#66

Post by binarycrusader » Wed, Mar 08, 2017, 12:10 AM UTC

codyps wrote: post #64

binarycrusader wrote: post #11

I prefer the way Solaris solved this problem: 1) first, by eliminating the need for a context switch for libc calls such as gettimeofday(), gethrtime(), etc. (there is no public/supported interface on Solaris for syscalls, so libc would be used) 2) by providing additional, specific interfaces with certain guarantees: https://docs.oracle.com/cd/E53394_01/html/E54766/get-sec-fro... This was accomplished by creating a s…

For those actually curious about the implementation on solaris/illumos, heres a quick rundown (from looking at current illumos source): - comm_page (usr/src/uts/i86pc/ml/comm_page.s) is literally a page in kernel memory with specific variables that is mapped (usr/src/uts/intel/ia32/os/comm_page_util.c) as user|read-only (to be passed to userspace, kernel mapping is normal data, AFAICT) - the mapped comm_page is inser…

So it isn't reading the time from this memory page, it's using TSC. In the case of CLOCK_REALTIME, corrections that are applied to TSC are read from this memory page (comm_page).

This summary only applies to Illumos. The Solaris implementation diverged significantly around build 167 (2011) long after the last OpenSolaris build Illumos was based on (build 147). It changed again significantly in 2015.

I believe Circonus contributed an alternate implementation that does some of the same things as Solaris in 2016:

https://www.circonus.com/2016/09/time-but-faster/

With that said, you are correct that whether or not it will read from a memory page instead depends on which interfaces you are using (e.g. get_hrusec()) and other subtle details.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#67

Post by mulmen » Wed, Mar 08, 2017, 12:18 AM UTC

woolly wrote: post #58

a_t48 wrote: post #26

Earlier quoted context omitted.

Nitpick - `77 percent faster` is not the inverse of `77 percent slower`. The line that says `The results of this microbenchmark show that the vDSO method is about 77% faster` should read `446% faster`.

Should that not be 346% faster? If A takes 1 second and B takes two seconds, then B is 100% faster than A. So the calculation would be (B/A - 1) * 100. Applying this here gives around 346%. EDIT: B would, of course, take 100% longer than A, rather than be 100% faster.

How can something that takes twice as long be faster?

Re: Two frequently used system calls are ~77% slower on AWS EC2

#68

Post by Twirrim » Wed, Mar 08, 2017, 12:19 AM UTC

xenophonf wrote: post #16

Is this just an EC2 problem, or does it affect any Xen/KVM guest? I ran the test program on a Hyper-V VM running CentOS 7 and got the same result: 100 calls to the gettimeofday syscall. Conversely, I tested a vSphere guest (also running CentOS 7), which didn't call gettimeofday at all.

It depends on a number of things. My DigitalOcean instance has this problem. The virtual machine I spun up on Oracle's Bare Metal Cloud platform doesn't (disclaimer, I work for the team).

Re: Two frequently used system calls are ~77% slower on AWS EC2

#69

Post by wahern » Wed, Mar 08, 2017, 12:22 AM UTC

binarycrusader wrote: post #56

jdamato wrote: post #52

Earlier quoted context omitted.

Not quite. The vDSO provides a general syscall-wrapper mechanism for certain types of system call interfaces. It also provides implementations of gettimeofday clock_gettime and 2 other system calls completely in userland and acts precisely as you've described. Please see this[1] for a detailed explanation. For a shorter explanation, please see the vDSO man page[2]. Thanks for reading my blog post! [1]: https://blog.p…

I'm aware of the high level about VDSO implementation, but I would still say that the Solaris implementation is more narrowly focused and as a result does not have the subtle issues / tradeoffs that VDSO does. Also, I personally find VDSO disagreeable as do others although perhaps not in as dramatic terms as some: https://mobile.twitter.com/bcantrill/status/5548101655902617... I think Ian Lance Taylor's summary is th…

This all seems predicated on the fact that Solaris doesn't support direct system calls and the fact that they ship their kernel and libc as one unified whole (like BSDs). Solaris is free to update the layout of their shared data structures whenever they want[1].

Because Linux kernel interfaces are distinct and separate from libc, and given Linus' policy on backwards compatibility, Linux had two choices for an _interface_: 1) export a data structure to userland that could never change, or 2) export a code linking mechanism to userland that could never change. In that light the latter choice seems far more reasonable.

[1] The shared data structures for this particular feature. There are other kernel data structures that leak through the libc interface and for which Solaris is bound to maintain compatibility.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#70

Post by woolly » Wed, Mar 08, 2017, 12:30 AM UTC

mulmen wrote: post #67

woolly wrote: post #58

Earlier quoted context omitted.

Should that not be 346% faster? If A takes 1 second and B takes two seconds, then B is 100% faster than A. So the calculation would be (B/A - 1) * 100. Applying this here gives around 346%. EDIT: B would, of course, take 100% longer than A, rather than be 100% faster.

How can something that takes twice as long be faster?

You're right, of course: hadn't had the morning coffee. It should have been 'takes 100% longer' in the 1 second/2 seconds example. The point I was trying to make is that you have to factor in the initial 100% which doesn't contribute to the final value.