Live data from Hacker News

Two frequently used system calls are ~77% slower on AWS EC2

blog.packagecloud.io

81–90 of 101 posts

Re: Two frequently used system calls are ~77% slower on AWS EC2

#81
post #5

Author here, greetings. Anyone who finds this interesting may also enjoy our writeup describing every Linux system call method in detail [1]. [1]: https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g...

This is rather out of date. Everything works quite similarly, but the kernel code is very different these days.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#82

Earlier quoted context omitted.

You just described the old method Linux used that was vulnerable to info leaks iirc and why it now a vDSO

The Solaris method doesn't have the problem the other implementation did.

How does solaris find the page? If it's mapped to a fixed address then it does have that problem.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#83

Earlier quoted context omitted.

The Solaris method doesn't have the problem the other implementation did.

How does solaris find the page? If it's mapped to a fixed address then it does have that problem.

The default is to map the shared page to a randomized, available address within the process space.

libc gets the address of the page by looking it up in an auxiliary vector table that belongs to the process.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#84

Does anyone have any intuition around how this affects a variety of typical workflows? I imagine that these two syscalls are disproportionally likely to affect benchmarks more than real-world usage. How many times is this syscall happening on a system doing things like serving HTTP, or running batch jobs, or hosting a database, etc?

You can use strace and see!

Go to your staging environment, use `strace -f -c -p $PID -e trace=clock_gettime` (or don't use -p and just launch the binary directly), replay a bit of production traffic against it, and then interrupt it and check the summary.

HTTP servers typically return a date header, often internally dates are used to figure out expiration and caching, and logging almost always includes dates.

It's incredibly easy to check the numbers of syscalls with strace, so you really should be able to get an intuition fairly easily by just playing around in staging.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#85
post #4

Another option is to reduce usage of gettimeofday() when possible. It is not always free. Roughly 10 years ago, when I was the driver author for one of the first full-speed 10GbE NICs, we'd get complaints from customers that were sure our NIC could not do 10Gbs, as iperf showed it was limited to 3Gb/s or less. I would ask them to re-try with netperf, and they'd see full bandwidth. I eventually figured out that the co…

> Another option is to reduce usage of gettimeofday() when possible. It is not always free. haha, it's amazing how much software is written that basically does something ridiculous like "while (gettimeofday()) clock_gettime();".

I found the articles on your blog about that topic quite interesting.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#86
If anybody is interested, Google Compute Engine VM's result.

    blog   ~ touch test.c
    blog   ~ nano test.c
    blog   ~ gcc -o test test.c
    blog   ~ strace -ce gettimeofday ./test
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      0.00    0.000000           0       100           gettimeofday
    ------ ----------- ----------- --------- --------- ----------------
    100.00    0.000000

Re: Two frequently used system calls are ~77% slower on AWS EC2

#88
> All programmers deploying software to production environments should regularly strace their applications in development mode and question all output they find.

Or, instead, you could just not do that. Then you could go back to being productive, instead of wasting time tracking down unstable small tweaks for edge cases that you can barely notice after looping the same syscall 5 million times in a row.

When will people learn not to micro-optimize?

Re: Two frequently used system calls are ~77% slower on AWS EC2

#89
post #87

Wasn't a workaround posted for this some time ago, that requires setting the TZ environment variable? https://news.ycombinator.com/item?id=13697555 It seems very closely related, unless I am mistaken.

You are not mistaken in that the topics are (somewhat) related, they all have to do with time. But setting the TZ environment variable doesn't mean your programs don't execute the syscalls discussed in this article.

This is about the speed of execution of the mentioned syscalls, which will be called regardless of the TZ environment variable, and how vDSO changes that. However, by setting the TZ environment variable you can avoid an additional call to stat to as it tries to determine if /etc/localtime exists.

Re: Two frequently used system calls are ~77% slower on AWS EC2

#90
post #4

Another option is to reduce usage of gettimeofday() when possible. It is not always free. Roughly 10 years ago, when I was the driver author for one of the first full-speed 10GbE NICs, we'd get complaints from customers that were sure our NIC could not do 10Gbs, as iperf showed it was limited to 3Gb/s or less. I would ask them to re-try with netperf, and they'd see full bandwidth. I eventually figured out that the co…

Syscalls in general far too often gets treated as if they're as cheap as function calls, with people often never profiling to see just how much they can affect throughput.

Apart from gettimeofday() other "favourites" of mine that people are often blind to include apps that do lots of unnecessary stat-ing of files, as well as tiny read()/write()'s instead of buffering in userspace.

Post reply on HN