Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two frequently used system calls are ~77% slower on AWS EC2

blog.packagecloud.io

Post reply on HN
Print view View original

11–20 of 101 posts

Re: Two frequently used system calls are ~77% slower on AWS EC2

#11

Post by binarycrusader »

I prefer the way Solaris solved this problem:

1) first, by eliminating the need for a context switch for libc calls such as gettimeofday(), gethrtime(), etc. (there is no public/supported interface on Solaris for syscalls, so libc would be used)

2) by providing additional, specific interfaces with certain guarantees:

https://docs.oracle.com/cd/E53394_01/html/E54766/get-sec-fro...

This was accomplished by creating a shared page in which the time is updated in the kernel in a page that is created during system startup. At process exec time that page is mapped into every process address space.

Solaris' libc was of course updated to simply read directly from this memory page. Of course, this is more practical on Solaris because libc and the kernel are tightly integrated, and because system calls are not public interfaces, but this seems greatly preferable to the VDSO mechanism.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#13

Post by jdamato »

binarycrusader wrote: post #11

I prefer the way Solaris solved this problem: 1) first, by eliminating the need for a context switch for libc calls such as gettimeofday(), gethrtime(), etc. (there is no public/supported interface on Solaris for syscalls, so libc would be used) 2) by providing additional, specific interfaces with certain guarantees: https://docs.oracle.com/cd/E53394_01/html/E54766/get-sec-fro... This was accomplished by creating a s…

This is precisely what the vDSO does. The clocksources mentioned explicitly list themselves as not supporting this action, hence the fallback to a regular system call.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#14

Post by aluminussoma »

brendangregg wrote: post #10

Yes, this is why we (Netflix) default to tsc over the xen clocksource. I found the xen clocksource had become a problem a few years ago, quantified using flame graphs, and investigated using my own microbenchmark. Summarized details here: https://www.slideshare.net/brendangregg/performance-tuning-e...

Can you share if you needed to do anything to deal with time drift issues when using tsc? For my own systems, incorrect timestamps would cause a lot of issues.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#16

Post by xenophonf »

Is this just an EC2 problem, or does it affect any Xen/KVM guest?

I ran the test program on a Hyper-V VM running CentOS 7 and got the same result: 100 calls to the gettimeofday syscall. Conversely, I tested a vSphere guest (also running CentOS 7), which didn't call gettimeofday at all.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#17

Post by coldtea »

sly010 wrote: post #8
LanceH wrote: post #6

Earlier quoted context omitted.

Maybe you could set up some caching.

For time?

Yes, for time too. For one, if you don't need over second precision, they why have some of your servers e.g. ask for the current time thousands of times per second? There are ways to get a soft expiration that don't involve asking for the time.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#18

Post by beagle3 »

binarycrusader wrote: post #11

I prefer the way Solaris solved this problem: 1) first, by eliminating the need for a context switch for libc calls such as gettimeofday(), gethrtime(), etc. (there is no public/supported interface on Solaris for syscalls, so libc would be used) 2) by providing additional, specific interfaces with certain guarantees: https://docs.oracle.com/cd/E53394_01/html/E54766/get-sec-fro... This was accomplished by creating a s…

Sounds like the clock resolution would be limited to the ticket interrupt in this case - how does it handle high resolution timers?

Re: Two frequently used system calls are ~77% slower on AWS EC2

#20

Post by apetresc »

Does anyone have any intuition around how this affects a variety of typical workflows? I imagine that these two syscalls are disproportionally likely to affect benchmarks more than real-world usage. How many times is this syscall happening on a system doing things like serving HTTP, or running batch jobs, or hosting a database, etc?
Post reply on HN