Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How setting the TZ environment variable avoids thousands of system calls

blog.packagecloud.io

Post reply on HN
Print view View original

111–120 of 148 posts

Re: How setting the TZ environment variable avoids thousands of system calls

#111

Post by noselasd »

cbsmith wrote: post #96
noselasd wrote: post #61

Earlier quoted context omitted.

There's no usable mechanisms glibc can use for watching /etc/localtime for changes that does not mess up the program if it also decides to use any file watching features.

At least on key platforms, it's pretty easy to use an event driven model and watch for updates. Hell, if the vDSO handled TZ it'd be no problem.

I'm not saying it's impossible, but if you actually sit down and attempt to add this to glibc, you will come to the realization that it is not easy at all. If you want to e.g. use the inotify mechanism you have a few key decision points to make

* There's no place where glibc can run an event loop to watch for the changes, the application might not have an event loop.

* If you need to integrate with an event loop the application runs, it'll work fine as long as the user remembers to hook up the events and deal with the corner cases. You can use this approach without any special support for glibc, and set TZ env. variable yourself when /etc/localtime changes - though at the moment that will only work for single threaded programs (setenv()/putenv() is not thread safe)

* If you decide to run the event loop in a separate thread, you force every binary to be multi threaded and have quite a lot of corner cases to handle when fork()'ing.

* If you don't run an event loop, you're back to polling for changes, and hardly anything is gained.

* If you use the signal driven I/O notification mechanism, you interfer with the application use of signals and I/O notification, and also have a host of fork() corner cases to consider.

vDSO is not a magic silver bullet that can solve this, you would at least have to have the kernel manage timezone support, or perhaps better, have the ability to transparently manage arbitarily data, and then expose that through shared memory that vDSO can use. This will not happen anytime soon.

Re: How setting the TZ environment variable avoids thousands of system calls

#112

Post by mfukar »

tytso wrote: post #18

System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. I ran an experiment where I timed the runtime of the sample program provided in the OP, except I changed the number of calls to localtime() from ten times to a million. I then timed the difference with and without export TZ=:/etc/localhost. The net savings was .…

System calls in Linux are not faster than not doing them.

Re: How setting the TZ environment variable avoids thousands of system calls

#114

Post by deathanatos »

cbsmith wrote: post #56
tytso wrote: post #18

System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. I ran an experiment where I timed the runtime of the sample program provided in the OP, except I changed the number of calls to localtime() from ten times to a million. I then timed the difference with and without export TZ=:/etc/localhost. The net savings was .…

> System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. "fast" is a relative term, and is somewhat orthogonal to "efficient". There's a reason why certain functions use a vDSO. If you're just going to use a syscall anyway, there's kind of no point.

You're assuming that all cases where the vDSO call is made gets paired with a real syscall; that's simply not the case. There are plenty of calls in a server that won't need localtime (basically, anything that just needs the current time in UTC: best-practice code should not be looking at the machine's TZ setting¹). Look at the examples the article's author offers:

> formatting dates and times

This shouldn't require a call to localtime; more explanation on the part of the article is required here. Breaking a seconds-since-epoch out into year/mo/day/etc. is "simple" math, and shouldn't require a filesystem access. Something else is amiss here.

> for everything from log messages

You're about to hit disk; a cache'd stat() isn't going to matter.

> to SQL queries.

You're about to hit the network; a cache'd stat() isn't going to matter.

(Now, I'm not saying you shouldn't set TZ; if it saves some syscalls, fine, and it might be the only sane value anyways.)

¹one of my old teams had an informal rule that any invocation of datetime.datetime.now() was a bug.

Re: How setting the TZ environment variable avoids thousands of system calls

#115

Post by deathanatos »

vesinisa wrote: post #72
tytso wrote: post #18

System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. I ran an experiment where I timed the runtime of the sample program provided in the OP, except I changed the number of calls to localtime() from ten times to a million. I then timed the difference with and without export TZ=:/etc/localhost. The net savings was .…

> TZ=:/etc/localhost Hope this is just a typo in your comment, not the actual test ;)

This isn't a typo, but is part of the syntax used by the TZ variable. (The same format appears in the article itself.)

See `man timezone` on a Linux system[1]. Specifically, see the passage that I've quoted below. Note that this is the third of three different formats that the man page describes that you can use in TZ:

> The second format specifies that the timezone information should be read from a file:

    :[filespec]
> *If the file specification filespec is omitted, or its value cannot be interpreted, then Coordinated Universal Time (UTC) is used. If filespec is given, it specifies another tzfile(5)-format file to read the timezone information from. If filespec does not begin with a '/', the file specification is relative to the system timezone directory. If the colon is omitted each of the above TZ formats will be tried.

[1]: https://linux.die.net/man/3/timezone

Re: How setting the TZ environment variable avoids thousands of system calls

#116

Post by deathanatos »

kseistrup wrote: post #27

It would be highly inconvenient to have to set this variable if you live in a country where you change the timezone twice a year due to summertime.

Starting or stopping Daylight Saving Time is not a timezone change. A timezone is — roughly — a set of timekeeping rules that some set of people use. DST is just part of those timekeeping rules. That is, your timezone is not the same thing as your UTC offset.

For example, the timezone America/New_York contains information not only about the UTC offset, but the offset during and not during DST, and when DST starts and ends. (And historical starts and ends to, so that UTC → local conversions (and vice versa) use the rules that were present at that time, not the rules that are present now, which may be different.)

E.g., my home desktop runs in the timezone America/Los_Angeles all year around. Most of my servers run in the timezone UTC all year. Both always have the appropriately correct time.

Re: How setting the TZ environment variable avoids thousands of system calls

#117

Post by unwind »

rootbear wrote: post #15

Is there a reason why the path to the timezone file is prefixed with a colon? TZ=:/etc/localtime I've set TZ sometimes without the colon and it seem to work. I did a quick online search and didn't find anything relevant.

See https://news.ycombinator.com/item?id=13704054. The colon forces the file to be loaded, without it the other formats are tried first.

Re: How setting the TZ environment variable avoids thousands of system calls

#118

Post by vesinisa »

deathanatos wrote: post #115
vesinisa wrote: post #72

Earlier quoted context omitted.

> TZ=:/etc/localhost Hope this is just a typo in your comment, not the actual test ;)

This isn't a typo, but is part of the syntax used by the TZ variable. (The same format appears in the article itself.) See `man timezone` on a Linux system[1]. Specifically, see the passage that I've quoted below. Note that this is the third of three different formats that the man page describes that you can use in TZ: > The second format specifies that the timezone information should be read from a file: :[filespec]…

Sure, but at least on none of my Linux systems there is no such file /etc/localhost. I think the parent was referring to /etc/localtime. Not sure what is the behaviour if non-existent file is specified - perhaps the "value cannot be interpreted" case applies, but it's not pefectly clear, since it could be argued that the value is valid, just refers to a non-existent file.

Re: How setting the TZ environment variable avoids thousands of system calls

#119

Post by andrelaszlo »

tytso wrote: post #18

System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. I ran an experiment where I timed the runtime of the sample program provided in the OP, except I changed the number of calls to localtime() from ten times to a million. I then timed the difference with and without export TZ=:/etc/localhost. The net savings was .…

I did the same, but with 10M iterations:

    $ time ./tz     
    ./tz  2,24s user 6,28s system 98% cpu 8,612 total
    $ export TZ=:/etc/localtime
    $ time ./tz                
    ./tz  1,35s user 0,00s system 98% cpu 1,364 total
So 0.7 microseconds on my machine.

Re: How setting the TZ environment variable avoids thousands of system calls

#120

Post by kingosticks »

tytso wrote: post #18

System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. I ran an experiment where I timed the runtime of the sample program provided in the OP, except I changed the number of calls to localtime() from ten times to a million. I then timed the difference with and without export TZ=:/etc/localhost. The net savings was .…

I did the same experiment on a Raspberry Pi 2. The net saving was 5.803 seconds, so 5.803 microseconds per call.

Obviously if you care about performance then you wouldn't be running your program on a Raspberry Pi in the first place. But for everything else there's this free speed up.

Post reply on HN