Live data from Hacker News

C stdlib isn't threadsafe and even safe Rust didn't save us

edgedb.com

261–270 of 370 posts

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#261
post #232

Earlier quoted context omitted.

> It seems like the only reliable way to fix this is to change these functions so that they exclusively acquire a mutex. A mutex can ensure thread safety but risks deadlocks if not used carefully and will hurt performance...

Agree about performance, but wouldn't there need to be >1 mutex to risk a deadlock?

If it's not a "recursive mutex" (where you can call lock within the same thread on the same mutex more than once consecutively and it handled that), it's possible to lock on itself again (say in code which is recursive)...

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#262
post #96

Earlier quoted context omitted.

I have a different perspective: the underlying problem is calling setenv(). As far as I'm concerned, the environment is a read-only input parameter set on process creation like argv. It's not a mechanism for exchanging information within a process, as used here with SSL_CERT_FILE. And remember that the exec* family of calls has a version with an envp argument, which is what should be used if a child process is to be…

Environment variables are a gigantic, decades-old hack that nobody should be using... but instead everyone has rejected file-based configuration management and everyone is abusing environment variables to inject config into "immutable" docker containers...

We use env vars on cloud machines to hold various metadata information about the machines. They can be queried by any program and is extremely useful. It's too useful to be considered a hack. People just misuse them.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#263
post #226

Earlier quoted context omitted.

+1 on the "horrible idea" part. Thing is, the (history of the) UNIX APIs - call'em "libc" if you like - is littered with the undead corpses of horrible ideas. Who thought that having global file write offsets are great ? Append-only writes ? Global working directories ? The ability to write the password db via putpwent() ? Modifying your own envp or argv ? Why have a horribly-scaling hack like fcntl-based file lockin…

Append-only writes are actually amazing, having several processes writing into the same file and have their writes interleaved instead of destroying each other is almost impossible to re-create in the user space. And I still don't understand why processes "modifying their own envp or argv" are met with such revulsion in this comment thread except from the "I dislike that on ideological grounds" reason. Now, the abili…

I had to smirk at the sarcasm (intended or no). I merely included "processes modifying their env" amongst all these historical warts. I consider doing so as inevitably necessary as append writes, the advantages of which you aptly described. That's my opinion, underpinned by the history of those interfaces. I hope we can agree that the breakage is by-and-large in an (old, historical) interface that allows braindead usage, not in either the implementor or the user ?

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#264
From the backtrace, it seems strerror_r is not thread-safe, since it calls __dcigettext which calls getenv.

A similar bug related to setlocale was found in 2007 and fixed in 2014. That bug did not take getenv/setenv into account. https://sourceware.org/bugzilla/show_bug.cgi?id=5443

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#265
post #171

Earlier quoted context omitted.

But that won't actually fix the underlying problem, namely that getenv and setenv (or unsetenv, probably) cannot safely be called from different threads. It seems like the only reliable way to fix this is to change these functions so that they exclusively acquire a mutex.

But it would force Rust programs to add their own synchronization mechanism around them. As long as no two threads can call getenv/setenv at the same time then it’s fine.

The problem isn't something that Rust can solve.

The Rust stdlib is already using synchronization on the versions of these functions that are exposed from the Rust stdlib. That's why those functions were allowed to be marked as safe in the first place.

The problem is that people are calling C code from Rust (which already requires an unsafe annotation), and then that C code is doing silly thread-unsafe shenanigans for regrettable historical reasons.

It's beyond Rust's power to fix without cooperation from the underlying C code, which happens to be provided by the OS, which is just being compliant with Posix. Rust can only do so much when the platform itself is hell-bent on sabotaging you.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#266

Earlier quoted context omitted.

I'll take existence proofs [1] over personal insults but YMMV. You also may want to be careful assuming the expertise of people on this forum. Some people here are quite technical. [1] https://github.com/bminor/glibc/commit/7a61e7f557a97ab597d6f...

That isn't thread safe, it's safER. I am also quite technical, thanks.

[flagged]

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#267
post #4

In the Rust std, `set_var` and `remove_var` will correctly require using an `unsafe {}` block in the next edition (2024). The documentation does now mention the safety issue but obviously it was a mistake to make these functions safe originally (albeit a mistake even higher level languages have made). https://doc.rust-lang.org/stable/std/env/fn.set_var.html There is a patch for glibc which makes `getenv` safe in more…

Wow, glibc now

keep[s] older versions around and adopt[s] an exponential resizing policy. This results in an amortized constant space leak per active environment variable, but there already is such a leak for the variable itself (and that is even length-dependent, and includes no-longer used values).

There have got to be pathalogical uses out there where this will cause unbounded memory growth in well-formed (according to the API) programs, no?

Interesting to see this _introduce_ a ‘bug’ (unbounded memory growth) for these programs that follow the API in order to ‘fix’ programs that don’t (by using the API in multiple threads). Pragmatism over dogma I guess. Leaves me feeling a bit sketched out though.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#268

Earlier quoted context omitted.

Indeed, environment variables should be used to configure child processes, not to configure the current process, for non-shell programs, IMHO. Note that Java, and the JVM, doesn't allow changing environment variables. It was the right choice, even if painful at times.

I think there's a narrow window, at least in some programming languages, when environment variables can be set at the start of a process. But since it's global shared state, it needs to be write (0,1) and read many. No libraries should set them. No frameworks should set them, only application authors and it should be dead obvious to the entire team what the last responsible moment is to write an environment variable.…

I agree that libraries certainly should not. But why would writing be the right choice ever, even for applications? Doesn't it make far more sense to use env to create in some better-typed global configuration object, filling any gaps with defaults, then use that?

I'd go further and say env should always be read-only and libraries should never even read env vars.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#269
post #160
post #84

Earlier quoted context omitted.

Sure, there's a broader concept here, which doesn't require any operating system. But any alternate string->string map you define won't answer to C code calling getenv, won't be passed to child processes created with fork, won't be visible through /proc/$PID/environ, etc.

This is the context: > They did, it's called core. But it assumes no operating system at all, and environment variables require an operating system. I think there's some confusion here. The C standard library is an abstraction layer that exists to implement standard behavior on hardware. It's entirely unrelated to the existence of an OS. Things like "/proc/$PID/environ" have nothing to do with C. There are many stand…

I don't think I'm confused, but let's recapitulate the thread history as I understand it:

  Context: The setenv function is not thread-safe even in Rust
  Question: Why doesn't Rust implement a standard library without C?
  Answer: It does, but core lacks std::env, because env vars are part of an O/S
  Question: Is an O/S really necessary for env vars?
  Answer: Not conceptually, but without an O/S, env vars don't work as expected
I also like the sibling comment that pointed out env vars are social as much as technical. The key element is interoperability. And we haven't even discussed Windows, which has different functions and conventions for environment variables.

Now, let me address what you just said. First of all, on embedded, a freestanding C implementation is not even required to provide getenv at all. Second, while getenv is in standard C and required for hosted implementations, setenv is not. And the whole thread is really about setenv. Once we pull in setenv, we're talking not just about standard C but about POSIX, which is a specification for operating systems. I assume for the sake of fruitful discussion, we both accept that a variable put into the environment with setenv should be retrievable thereafter with getenv. This moreover should apply even if it's Rust that calls setenv and C that calls getenv and vice-versa.

So, however Rust implements environment variables should be consistent with how C implements environment variables, and since C provides the foundation for system calls and FFI for most other major languages, adhering to this convention allows interoperability across very many languages. This convention is defined by libc (the implementation of the C standard and POSIX interfaces) and thus interoperability is based on libc compatibility. So either Rust implements its own libc, which C programs would have to be (re-)compiled to use, or else it uses an existing implementation of libc, inheriting all of its quirks. Indeed, Rust targets specify the libc (or equivalent) they're using, such as -gnu, -musl, -darwin, -mingw, -msvc, etc. Linking with libraries built for a different libc on an otherwise identical platform (-gnu vs. -musl on Linux, -mingw vs -msvc on Windows) generally doesn't work and even when it appears to work leads to strange issues later. So you can't just write your own getenv and expect it to work with some other implementation of setenv.

To connect back with my other comments, there is no core::env because core assumes no libc at all. The nostd flavor of Rust (where core is available but not std) is basically equivalent to freestanding C and like freestanding C there is no interoperability guarantee, not even with freestanding C on the same hardware (indeed, the whole concept of "freestanding" is that there are no conventions to adhere to in the first place). So, std::env::set_env has the exact same problems as C setenv because it's the same thing under the hood. This cannot be addressed without fixing libc itself. Moreover, when libc is not involved, then there is no env to support to begin with.

Finally, to round out addressing what you said, core::env could exist, but probably shouldn't, for two reasons. First, it would be misleading. As I've already laid out, it would not interoperate with anything since there's nothing there to interoperate with. It would just be a global string->string map exclusive to that program, which the programmer could just as well create on his own. Second, because presumably you want it to be something other than empty, it would require some kind of global allocator, which core also assumes doesn't exist. So it would have to be something like alloc::env instead, and once you've pulled in alloc, you can just use one of the collection types (though, notably, HashMap isn't in alloc yet [1]).

[1]: https://github.com/rust-lang/rust/issues/27242

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#270
post #96

Earlier quoted context omitted.

I have a different perspective: the underlying problem is calling setenv(). As far as I'm concerned, the environment is a read-only input parameter set on process creation like argv. It's not a mechanism for exchanging information within a process, as used here with SSL_CERT_FILE. And remember that the exec* family of calls has a version with an envp argument, which is what should be used if a child process is to be…

Environment variables are a gigantic, decades-old hack that nobody should be using... but instead everyone has rejected file-based configuration management and everyone is abusing environment variables to inject config into "immutable" docker containers...

> instead everyone has rejected file-based configuration management

With good reason. Files are surprisingly hard: https://danluu.com/deconstruct-files/

Post reply on HN