Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two frequently used system calls are ~77% slower on AWS EC2

blog.packagecloud.io

Post reply on HN
Print view View original

71–80 of 101 posts

Re: Two frequently used system calls are ~77% slower on AWS EC2

#71

Post by stouset »

klodolph wrote: post #44
nneonneo wrote: post #36

The title is misleading. 77% slower sounds like the system calls take 1.77x the time on EC2. In fact, the results indicate that the normal calls are 77% faster - in other words, EC2 gettimeofday and clock_gettime calls take nearly 4.5x longer to run on EC2 than they do on ordinary systems. This is a big speed hit. Some programs can use gettimeofday extremely frequently - for example, many programs call timing functio…

77% faster is not correct either. "Speed" would probably by ops/s. 4.5x longer = 350% slower.

Even this is confusing as hell.

Just say the native calls take 22% of the time they do on EC2. Or that the EC2 calls take 450% of the time of their native counterparts.

"Faster" and "slower" when going with percentages are ripe with confusion. Please don't use them.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#72

Post by kzrdude »

derefr wrote: post #55
nneonneo wrote: post #36

The title is misleading. 77% slower sounds like the system calls take 1.77x the time on EC2. In fact, the results indicate that the normal calls are 77% faster - in other words, EC2 gettimeofday and clock_gettime calls take nearly 4.5x longer to run on EC2 than they do on ordinary systems. This is a big speed hit. Some programs can use gettimeofday extremely frequently - for example, many programs call timing functio…

> Some programs can use gettimeofday extremely frequently This is what's usually considered the "root cause" of this problem, though. It's easy enough, if it's your own program, to wrap the OS time APIs to cache the evaluated timestamp for one event-loop (or for a given length of realtime by checking with the TSC.) Most modern interpreters/VM runtimes also do this.

Why cache it when the vdso has already solved the problem? Seems best to not stack another mitigation on top of it.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#73

Post by codyps »

binarycrusader wrote: post #66
codyps wrote: post #64

Earlier quoted context omitted.

For those actually curious about the implementation on solaris/illumos, heres a quick rundown (from looking at current illumos source): - comm_page (usr/src/uts/i86pc/ml/comm_page.s) is literally a page in kernel memory with specific variables that is mapped (usr/src/uts/intel/ia32/os/comm_page_util.c) as user|read-only (to be passed to userspace, kernel mapping is normal data, AFAICT) - the mapped comm_page is inser…

So it isn't reading the time from this memory page, it's using TSC. In the case of CLOCK_REALTIME, corrections that are applied to TSC are read from this memory page (comm_page). This summary only applies to Illumos. The Solaris implementation diverged significantly around build 167 (2011) long after the last OpenSolaris build Illumos was based on (build 147). It changed again significantly in 2015. I believe Circonu…

So the only things I'm seeing in the linked circonus code that differ from illumos:

1. no use of a kernel supplied page, determines skew/etc itself in userspace 2. stores information on a per-cpu level, and tries to execute cpuid on the same cpu as rdtsc.

I'm presuming you're talking about #2 (and #1 is just due to the linked item being a library without kernel integrations)? Perhaps with some more kernel support so that the actual cpu rdtsc ran on can be reliably determined?

This still doesn't clarify the part about "shared page in which the time is updated" and is read from. This statement appears to imply TSC is not (necessarily) used (otherwise I'd categorize it under "uses values from memory page to fixup TSC", like Illumos' current implimentation). I'm still not sure how that can be done reasonably.

Is there just a 1 micro second timer running whenever a user task is being executed that is bumping the value? Wouldn't that be quite a bit of overhead? Or some HW trick? I mean, you could generate a fault on every read, and have the kernel populate the current data, but that seems just as bad as a syscall.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#74

Post by OJFord »

throwaway2048 wrote: post #60
monktastic1 wrote: post #15

Earlier quoted context omitted.

Yeah, with a TTL, and on each round you just check the time to see if it's expired.

yeah just call gettimeofday() to see if it expired yet.

Obviously you don't use clock-TTL.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#75

Post by derefr »

kzrdude wrote: post #72
derefr wrote: post #55

Earlier quoted context omitted.

> Some programs can use gettimeofday extremely frequently This is what's usually considered the "root cause" of this problem, though. It's easy enough, if it's your own program, to wrap the OS time APIs to cache the evaluated timestamp for one event-loop (or for a given length of realtime by checking with the TSC.) Most modern interpreters/VM runtimes also do this.

Why cache it when the vdso has already solved the problem? Seems best to not stack another mitigation on top of it.

Because you're writing portable code and not every (or even most) systems do said caching.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#76

Post by caf »

binarycrusader wrote: post #46
jdamato wrote: post #13

Earlier quoted context omitted.

This is precisely what the vDSO does. The clocksources mentioned explicitly list themselves as not supporting this action, hence the fallback to a regular system call.

Not quite; vdso is a general syscall-wrapper mechanism. The Solaris solution is specifically just for the gettimeofday(), gethrtime() interfaces, etc. The difference is that on Solaris, since there is no public system call interface, there's also no need for a fallback. Every program is just faster, no matter how Solaris is virtualized, since every program is using libc. There's also no need for an administrative int…

The fallback isn't there because there's a public system call interface: the fallback is there because some of the kernel-side implementations of gettimeofday() (in particular, the Xen one) currently require the process to do a proper syscall.

This is separate from the fact that the gettimeofday() system call still exists too, which is a backwards-compatibility issue. The overwhelming majority of Linux applications do their system calls through libc too, so this doesn't affect them.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#77

Post by a_t48 »

woolly wrote: post #58
a_t48 wrote: post #26

Earlier quoted context omitted.

Nitpick - `77 percent faster` is not the inverse of `77 percent slower`. The line that says `The results of this microbenchmark show that the vDSO method is about 77% faster` should read `446% faster`.

Should that not be 346% faster? If A takes 1 second and B takes two seconds, then B is 100% faster than A. So the calculation would be (B/A - 1) * 100. Applying this here gives around 346%. EDIT: B would, of course, take 100% longer than A, rather than be 100% faster.

Yes. Foot meet mouth. :)

Re: Two frequently used system calls are ~77% slower on AWS EC2

#80

Post by amluto »

binarycrusader wrote: post #56
jdamato wrote: post #52

Earlier quoted context omitted.

Not quite. The vDSO provides a general syscall-wrapper mechanism for certain types of system call interfaces. It also provides implementations of gettimeofday clock_gettime and 2 other system calls completely in userland and acts precisely as you've described. Please see this[1] for a detailed explanation. For a shorter explanation, please see the vDSO man page[2]. Thanks for reading my blog post! [1]: https://blog.p…

I'm aware of the high level about VDSO implementation, but I would still say that the Solaris implementation is more narrowly focused and as a result does not have the subtle issues / tradeoffs that VDSO does. Also, I personally find VDSO disagreeable as do others although perhaps not in as dramatic terms as some: https://mobile.twitter.com/bcantrill/status/5548101655902617... I think Ian Lance Taylor's summary is th…

> Not quite; vdso is a general syscall-wrapper mechanism.

It's not. On 32-bit x86, it sort of is, but that's just because the 32-bit x86 fast syscall mechanism isn't really compatible with inline syscalls. Linux (and presumably most other kernels) provides a wrapper function that means "do a syscall". It's only accelerated insofar as it uses a faster hardware mechanism. It has nothing to do with fast timing.

On x86_64, there is no such mechanism.

> It's true that dynamically linked programs can use the ELF loader. But the ELF loader needed special changes to support VDSOs. And so did gdb. And this approach doesn't help statically linked programs much.

That's because the glibc ELF loader is a piece of, ahem, is baroque and overcomplicated. And there's no reason whatsoever that vDSO usage needs to be integrated with the dynamic linker at all.

I wrote a CC0-licensed standalone vDSO parser here:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....

It's 269 lines of code, including lots of comments, and it works in static binaries just fine. Go's runtime (which is static!) uses a vDSO loader based on it. I agree that a static table would be slightly simpler, but the tooling for debugging the vDSO is a heck of a lot simpler with the ELF approach.

Post reply on HN