Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two frequently used system calls are ~77% slower on AWS EC2

blog.packagecloud.io

Post reply on HN
Print view View original

51–60 of 101 posts

Re: Two frequently used system calls are ~77% slower on AWS EC2

#51

Post by pgaddict »

blibble wrote: post #50
pgaddict wrote: post #45

Earlier quoted context omitted.

Not really. Recent CPUs (at least those from Intel, which is what EC2 runs on) implement constant_tsc, so the frequency does not affect the tsc. A worse issue is that the counters may not be synchronized between cpus, which may be an issue when the process moves between sockets. But I wouldn't call that "dangerous", it's simply a feature of the clock source. If that's an issue for your program, you should use CLOCK_M…

how does constant_tsc interact with VMs being silently migrated from one physical machine to another?

Not sure, but it can't be better than moving processes between CPUs I guess. Also, does EC2 silently move VMs like this?

Re: Two frequently used system calls are ~77% slower on AWS EC2

#52

Post by jdamato »

binarycrusader wrote: post #46
jdamato wrote: post #13

Earlier quoted context omitted.

This is precisely what the vDSO does. The clocksources mentioned explicitly list themselves as not supporting this action, hence the fallback to a regular system call.

Not quite; vdso is a general syscall-wrapper mechanism. The Solaris solution is specifically just for the gettimeofday(), gethrtime() interfaces, etc. The difference is that on Solaris, since there is no public system call interface, there's also no need for a fallback. Every program is just faster, no matter how Solaris is virtualized, since every program is using libc. There's also no need for an administrative int…

Not quite. The vDSO provides a general syscall-wrapper mechanism for certain types of system call interfaces. It also provides implementations of gettimeofday clock_gettime and 2 other system calls completely in userland and acts precisely as you've described.

Please see this[1] for a detailed explanation. For a shorter explanation, please see the vDSO man page[2]. Thanks for reading my blog post!

[1]: https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g... [2]: http://man7.org/linux/man-pages/man7/vdso.7.html

Re: Two frequently used system calls are ~77% slower on AWS EC2

#53

Post by jdamato »

pgaddict wrote: post #51
blibble wrote: post #50

Earlier quoted context omitted.

how does constant_tsc interact with VMs being silently migrated from one physical machine to another?

Not sure, but it can't be better than moving processes between CPUs I guess. Also, does EC2 silently move VMs like this?

We don't really know what EC2 does or precisely the type of hardware your VM will be spun up on. I've erred on the side of being cautious due to the vast amount of work being invested in timekeeping in various hypervisors. If EC2 knows that the TSC clocksource is safe on all of its hardware, perhaps modifying the Amazon Linux AMI to set TSC as the default clocksource would reassure many folks, myself included.

Advanced users that can run their own analysis or who have applications which would withstand potential time warps, are of course, free to ignore my warning at their own risk ;)

Re: Two frequently used system calls are ~77% slower on AWS EC2

#54

Post by kondro »

blibble wrote: post #50
pgaddict wrote: post #45

Earlier quoted context omitted.

Not really. Recent CPUs (at least those from Intel, which is what EC2 runs on) implement constant_tsc, so the frequency does not affect the tsc. A worse issue is that the counters may not be synchronized between cpus, which may be an issue when the process moves between sockets. But I wouldn't call that "dangerous", it's simply a feature of the clock source. If that's an issue for your program, you should use CLOCK_M…

how does constant_tsc interact with VMs being silently migrated from one physical machine to another?

EC2 doesn't do any kind of live machine migration. The only times a machine may start on a different host is if it is stopped and then started. Even reboots don't allow them to move.

You see this a lot when AWS lets you know about maintenance on a physical host and gives you the option to avoid the automated move by doing these steps manually at a time of your choosing before the maintenance window.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#55

Post by derefr »

nneonneo wrote: post #36

The title is misleading. 77% slower sounds like the system calls take 1.77x the time on EC2. In fact, the results indicate that the normal calls are 77% faster - in other words, EC2 gettimeofday and clock_gettime calls take nearly 4.5x longer to run on EC2 than they do on ordinary systems. This is a big speed hit. Some programs can use gettimeofday extremely frequently - for example, many programs call timing functio…

> Some programs can use gettimeofday extremely frequently

This is what's usually considered the "root cause" of this problem, though. It's easy enough, if it's your own program, to wrap the OS time APIs to cache the evaluated timestamp for one event-loop (or for a given length of realtime by checking with the TSC.) Most modern interpreters/VM runtimes also do this.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#56

Post by binarycrusader »

jdamato wrote: post #52
binarycrusader wrote: post #46

Earlier quoted context omitted.

Not quite; vdso is a general syscall-wrapper mechanism. The Solaris solution is specifically just for the gettimeofday(), gethrtime() interfaces, etc. The difference is that on Solaris, since there is no public system call interface, there's also no need for a fallback. Every program is just faster, no matter how Solaris is virtualized, since every program is using libc. There's also no need for an administrative int…

Not quite. The vDSO provides a general syscall-wrapper mechanism for certain types of system call interfaces. It also provides implementations of gettimeofday clock_gettime and 2 other system calls completely in userland and acts precisely as you've described. Please see this[1] for a detailed explanation. For a shorter explanation, please see the vDSO man page[2]. Thanks for reading my blog post! [1]: https://blog.p…

I'm aware of the high level about VDSO implementation, but I would still say that the Solaris implementation is more narrowly focused and as a result does not have the subtle issues / tradeoffs that VDSO does.

Also, I personally find VDSO disagreeable as do others although perhaps not in as dramatic terms as some:

https://mobile.twitter.com/bcantrill/status/5548101655902617...

I think Ian Lance Taylor's summary is the most balanced and thoughtful:

Basically you want the kernel to provide a mapping for a small number of magic symbols to addresses that can be called at runtime. In other words, you want to map a small number of indexes to addresses. I can think of many different ways to handle that in the kernel. I don't think the first mechanism I would reach for would be for the kernel to create an in-memory shared library. It's kind of a baroque mechanism for implementing a simple table.

It's true that dynamically linked programs can use the ELF loader. But the ELF loader needed special changes to support VDSOs. And so did gdb. And this approach doesn't help statically linked programs much. And glibc functions needed to be changed anyhow to be aware of the VDSO symbols. So as far as I can tell, all of this complexity really didn't get anything for free. It just wound up being complex.

All just my opinion, of course.

https://github.com/golang/go/issues/8197#issuecomment-660959...

Re: Two frequently used system calls are ~77% slower on AWS EC2

#57

Post by poofyleek »

pgaddict wrote: post #51
blibble wrote: post #50

Earlier quoted context omitted.

how does constant_tsc interact with VMs being silently migrated from one physical machine to another?

Not sure, but it can't be better than moving processes between CPUs I guess. Also, does EC2 silently move VMs like this?

Even without migration, the synchronization can be an issue. In older multi-core machines, tsc synchronization was an issue among cores. Modern systems take care of this. And core CPU clock frequency change is also taken care of, so that constant rate is available via tsc. However, when hypervisors such as VMWare or paravirtualization like Xen come into play, there are further issues, because RDTSC instruction either has to be passed through to physical hardware or emulated via a trap. When emulated a number of considerations come into play. Xen actually has PVRDTSC features that are normally not used but can be effective in paravirtual environments. The gettimeofday() syscalls (and clock_gettime) are liberally used in too many lines of existing software. Their use is very prevalent due to historical reasons as well as many others. One reason is that the calls are deceptively "atomic" or "isolated" or "self-contained" in their appearance and usage. So liberal use is common. A lot of issues come about due to their use, especially in time sensitive applications (e.g. WAN optimization). This is especially true in virtual environments. There are complex issues described elsewhere that are kind of fun to read. https://www.vmware.com/pdf/vmware_timekeeping.pdf and https://xenbits.xen.org/docs/4.3-testing/misc/tscmode.txt. The issue becomes even more complex in distributed systems. Beyond NTP. Some systems like erlang has some provisions, like http://erlang.org/doc/apps/erts/time_correction.html#OS_Syst.... Other systems use virtual vector clocks. And some systems, like google TrueTime as used in Spanner, synchronize using GPS atomic clocks. The satellite GPS pulses are commonly used in trading floors and HFT software. This is a very interesting area of study.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#58

Post by woolly »

a_t48 wrote: post #26
jdamato wrote: post #5

Author here, greetings. Anyone who finds this interesting may also enjoy our writeup describing every Linux system call method in detail [1]. [1]: https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g...

Nitpick - `77 percent faster` is not the inverse of `77 percent slower`. The line that says `The results of this microbenchmark show that the vDSO method is about 77% faster` should read `446% faster`.

Should that not be 346% faster? If A takes 1 second and B takes two seconds, then B is 100% faster than A. So the calculation would be (B/A - 1) * 100. Applying this here gives around 346%.

EDIT: B would, of course, take 100% longer than A, rather than be 100% faster.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#59

Post by pgaddict »

poofyleek wrote: post #57
pgaddict wrote: post #51

Earlier quoted context omitted.

Not sure, but it can't be better than moving processes between CPUs I guess. Also, does EC2 silently move VMs like this?

Even without migration, the synchronization can be an issue. In older multi-core machines, tsc synchronization was an issue among cores. Modern systems take care of this. And core CPU clock frequency change is also taken care of, so that constant rate is available via tsc. However, when hypervisors such as VMWare or paravirtualization like Xen come into play, there are further issues, because RDTSC instruction either…

It's complex stuff, no doubt about that.

For me, it's much simpler - I come from the PostgreSQL world, so gettimeofday() is pretty much what EXPLAIN ANALYZE does to instrument queries. Good time source means small overhead, bad time source means instrumented queries may take multiples of actual run time (and be skewed in various ways). No fun.

Post reply on HN