Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

C stdlib isn't threadsafe and even safe Rust didn't save us

edgedb.com

Post reply on HN
Print view View original

171–180 of 370 posts

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#171

Post by db48x »

usefulcat wrote: post #77
mmastrac wrote: post #5

The major takeaway from this is that Rust will be making environment setters unsafe in the next edition. With luck, this will filter down into crates that trigger these crashes ( https://github.com/alexcrichton/openssl-probe/issues/30 filed upstream in the meantime).

But that won't actually fix the underlying problem, namely that getenv and setenv (or unsetenv, probably) cannot safely be called from different threads. It seems like the only reliable way to fix this is to change these functions so that they exclusively acquire a mutex.

But it would force Rust programs to add their own synchronization mechanism around them. As long as no two threads can call getenv/setenv at the same time then it’s fine.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#172

Post by dietr1ch »

hinkley wrote: post #152
diroussel wrote: post #107

Earlier quoted context omitted.

Indeed, environment variables should be used to configure child processes, not to configure the current process, for non-shell programs, IMHO. Note that Java, and the JVM, doesn't allow changing environment variables. It was the right choice, even if painful at times.

I think there's a narrow window, at least in some programming languages, when environment variables can be set at the start of a process. But since it's global shared state, it needs to be write (0,1) and read many. No libraries should set them. No frameworks should set them, only application authors and it should be dead obvious to the entire team what the last responsible moment is to write an environment variable.…

Which programming languages?

When using C++ I wanted programs to have a function that was called before main() and set up things that got sealed afterwards, like parsing command-line-arguments, the environment variables, loading runtime libraries, and maybe look at the local directory, but I'm not sure if it'll be a useful and meaningful distinction unless you restructure way too many things.

I remember that on the Fuchsia kernel programs needed to drop capabilities at some point, but the shift needed might be a hard sell given things already "work fine".

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#173

Post by zanderwohl »

IshKebab wrote: post #57
zanderwohl wrote: post #26

Earlier quoted context omitted.

It would be a tremendous amount of work, and would take years. Meanwhile, the problems are avoidable. It's not exactly the "rust way" to just remember and avoid problems, but everything in language design is compromises.

"Impossibru!!" https://github.com/sunfishcode/eyra Oh look: > Why use Eyra? It fixes Rust's set_var unsoundness issue. The environment-variable implementation leaks memory internally (it is optional, but enabled by default), so setenv etc. are thread-safe.

> Why not use Eyra?

Well, that's a lot of caveats. As I said, it would take years to complete. And it looks like it's well on its way but not near complete.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#174

Post by harrall »

stouset wrote: post #166
eqvinox wrote: post #96

Earlier quoted context omitted.

I have a different perspective: the underlying problem is calling setenv(). As far as I'm concerned, the environment is a read-only input parameter set on process creation like argv. It's not a mechanism for exchanging information within a process, as used here with SSL_CERT_FILE. And remember that the exec* family of calls has a version with an envp argument, which is what should be used if a child process is to be…

> As far as I'm concerned, the environment is a read-only input parameter set on process creation like argv. This holds for a lot of programs, but what if you're writing a shell?

I’ve written a lot of subprocess runners and environmental variables passed to a sub-process is just data at that point and you store it in your own variable like you would store someone’s name or someone’s age.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#175

Post by o11c »

tsimionescu wrote: post #167
debugnik wrote: post #132

Earlier quoted context omitted.

No amount of locking can make the getenv API thread-safe, because it returns a pointer which gets invalidated by setenv, but lacks a way to release ownership over it and unblock setenv safely (or to free a returned copy). So setenv's existence makes getenv inherently unsafe unless you can ensure the entire application is at a safe point to use them.

This is actually not that hard to fix. Getenv() could keep several copies of the value around: one internal copy protected by a mutex, that it never returns, and one copy per thread that it stores in thread local storage. When you call getenv(), it locks the mutex, checks if the current thread's value exists, populates it from the internal copy if not, and returns it. It will also install a new setenv-specific signal…

If you only consider `getenv`/`setenv` there are indeed many solutions, but it's not that simple. You also need to consider `putenv` (not that nasty, you just need to treat it like initial environment, which means you can't use a single range check) and accessing the `environ` variable directly (nasty).

Your particular solution doesn't work because people expect `getenv` to be async-signal-safe, which means you shouldn't be allocating memory.

Hmm ... doing an incref-like operation during `getenv` for a previously `setenv`ed variable that hasn't yet been accessed in this thread would be fine ... clear those refs during calls we know indicate knowledge refreshes ...

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#176

Post by sunshowers »

tsimionescu wrote: post #165
sunshowers wrote: post #102

Earlier quoted context omitted.

That is a technical solution. What is your solution to the much more serious social problem of adding this check to every codebase in existence? What points of leverage do you have?

The point was about adding a mutex inside libc in getenv and setenv. That way, every codebase in existence automatically gets this safety. The poster I was replying to claimed that this wouldn't help, because it would still not offer thread safety when doing multiple operations. I pointed out that, in addition to libc setenv/getenv using a mutex internally, they could also expose new functions to allow transactional…

Ah I see — in that case you might be interested in the lockless illumos impl https://github.com/illumos/illumos-gate/blob/master/usr/src/...

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#177

Post by IshKebab »

vlovich123 wrote: post #148
usefulcat wrote: post #113

Earlier quoted context omitted.

The biggest problem is not the absence of a thread safe API, it's the existence of this: extern char **environ; As long as environ is publicly accessible, there's no guarantee that setenv and getenv will be used at all, since they're not necessary. If you're willing to get rid of environ, it's pretty trivial to make setenv and getenv thread safe. If not, then it's impossible, although one could still argue that makin…

> aka don't let the perfect be the enemy of the good Exactly my point. Over time *environ would disappear, at least from the major software projects that everyone uses (assuming it's even in use in them in the first place).

Yeah I don't think I've ever seen a single use of it. However I just checked on grep.app and at least a few big softwares use it - git, nginx, Postgresql, neovim, etc, which suggests that setenv/getenv is not sufficient.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#178

Post by oguz-ismail »

o11c wrote: post #175
tsimionescu wrote: post #167

Earlier quoted context omitted.

This is actually not that hard to fix. Getenv() could keep several copies of the value around: one internal copy protected by a mutex, that it never returns, and one copy per thread that it stores in thread local storage. When you call getenv(), it locks the mutex, checks if the current thread's value exists, populates it from the internal copy if not, and returns it. It will also install a new setenv-specific signal…

If you only consider `getenv`/`setenv` there are indeed many solutions, but it's not that simple. You also need to consider `putenv` (not that nasty, you just need to treat it like initial environment, which means you can't use a single range check) and accessing the `environ` variable directly (nasty). Your particular solution doesn't work because people expect `getenv` to be async-signal-safe, which means you shoul…

>`putenv` (not that nasty,

It's equally nasty. POSIX requires that the argument to `putenv()' not be copied, so it's not very different from assigning to `environ' directly.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#179

Post by Ferret7446 »

usefulcat wrote: post #77
mmastrac wrote: post #5

The major takeaway from this is that Rust will be making environment setters unsafe in the next edition. With luck, this will filter down into crates that trigger these crashes ( https://github.com/alexcrichton/openssl-probe/issues/30 filed upstream in the meantime).

But that won't actually fix the underlying problem, namely that getenv and setenv (or unsetenv, probably) cannot safely be called from different threads. It seems like the only reliable way to fix this is to change these functions so that they exclusively acquire a mutex.

Is that a problem? I feel like calling getenv and setenv from different threads is a design antipattern anyway. Any environment setting and loading should happen in the one and only main thread right after process init.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#180

Post by lmm »

PaulDavisThe1st wrote: post #90
wmf wrote: post #14

Earlier quoted context omitted.

It's time to move beyond this attitude and make things safe by default. For example, Solaris has a safer version of setenv(). "It is ridiculous that this has been a known problem for so long. It has wasted thousands of hours of people's time, either debugging the problems, or debating what to do about it. We know how to fix the problem." https://www.evanjones.ca/setenv-is-not-thread-safe.html

One of the major differences between X Window and the win32 GUI APIs is that the windows one builds in thread safety, and it cannot be removed. This means that you pay the price of mutexes and the like (what the windows world likes to call "critical sections"), even if you have a single threaded GUI. X Window, on the other hand, decided to do nothing about threads at all, leaving it up to the application. 30 years af…

> Consequently, the overhead present in the win32 API is basically just dead-weight, there to make sure that "things are safe by default".

How much overhead is it though? IIRC uncontended mutexes are practically free, especially when they're only being used from a single thread.

Our industry is way too eager to make things unsafe for the sake of marginal performance differences that are irrelevant for most use cases, IMO.

Post reply on HN