Two frequently used system calls are ~77% slower on AWS EC2
blog.packagecloud.io
Two frequently used system calls are ~77% slower on AWS EC2
1–10 of 101 posts
Re: Two frequently used system calls are ~77% slower on AWS EC2
#2Previous related discussion: https://news.ycombinator.com/item?id=13697555
Re: Two frequently used system calls are ~77% slower on AWS EC2
#3So… it's not that the syscalls are slower, it's that the Linux-specific mechanism the Linux kernel uses to bypass having to actually perform these calls does not currently work on Xen (and thus EC2).
Re: Two frequently used system calls are ~77% slower on AWS EC2
#4Another option is to reduce usage of gettimeofday() when possible. It is not always free.
Roughly 10 years ago, when I was the driver author for one of the first full-speed 10GbE NICs, we'd get complaints from customers that were sure our NIC could not do 10Gbs, as iperf showed it was limited to 3Gb/s or less. I would ask them to re-try with netperf, and they'd see full bandwidth. I eventually figured out that the complaints were coming from customers running distros without the vdso stuff, and/or running other OSes which (at the time) didn't support that (Mac OS, FreeBSD). It turns out that the difference was that iperf would call gettimeofday() around every socket write to measure bandwidth. But netperf would just issue gettimeofday calls at the start and the end of the benchmark, so iperf was effectively gettimeofday bound. Ugh.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#5Author here, greetings. Anyone who finds this interesting may also enjoy our writeup describing every Linux system call method in detail [1].
[1]: https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g...
Re: Two frequently used system calls are ~77% slower on AWS EC2
#6Another option is to reduce usage of gettimeofday() when possible. It is not always free. Roughly 10 years ago, when I was the driver author for one of the first full-speed 10GbE NICs, we'd get complaints from customers that were sure our NIC could not do 10Gbs, as iperf showed it was limited to 3Gb/s or less. I would ask them to re-try with netperf, and they'd see full bandwidth. I eventually figured out that the co…
Maybe you could set up some caching.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#7Another option is to reduce usage of gettimeofday() when possible. It is not always free. Roughly 10 years ago, when I was the driver author for one of the first full-speed 10GbE NICs, we'd get complaints from customers that were sure our NIC could not do 10Gbs, as iperf showed it was limited to 3Gb/s or less. I would ask them to re-try with netperf, and they'd see full bandwidth. I eventually figured out that the co…
Maybe you could set up some caching.
Caching is basically what the vdso things do. In my recollection, they basically grab a good time from the kernel occasionally, and then use userspace accessible things like rdtsc() to offset from that authoritative timestamp. So it turns millions of syscalls into one.
Re: Two frequently used system calls are ~77% slower on AWS EC2
#8Another option is to reduce usage of gettimeofday() when possible. It is not always free. Roughly 10 years ago, when I was the driver author for one of the first full-speed 10GbE NICs, we'd get complaints from customers that were sure our NIC could not do 10Gbs, as iperf showed it was limited to 3Gb/s or less. I would ask them to re-try with netperf, and they'd see full bandwidth. I eventually figured out that the co…
Maybe you could set up some caching.
For time?
Re: Two frequently used system calls are ~77% slower on AWS EC2
#9Re: Two frequently used system calls are ~77% slower on AWS EC2
#10Yes, this is why we (Netflix) default to tsc over the xen clocksource. I found the xen clocksource had become a problem a few years ago, quantified using flame graphs, and investigated using my own microbenchmark.
Summarized details here:
https://www.slideshare.net/brendangregg/performance-tuning-e...