Live data from Hacker News

C stdlib isn't threadsafe and even safe Rust didn't save us

edgedb.com

201–210 of 370 posts

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#201
post #191

Earlier quoted context omitted.

x20 is a general purpose register; optimizing compilers can use it for any number of variables, immediate values or intermediate computations at different points within that same function — or none at all (the variable ep could be optimized away). Re: fork(), I just meant to be thorough in explaining the environment is copied, not shared by processes. Setenv() only affects the process from which it’s called. The arra…

No, it does not. I don't think you understand what you are talking about, because none of these actually address the points I brought up. They use the same words, but semantically they are talking about something completely different.

I provided a copy/paste from the site about the envp array size you asked about.

I clarified why I mentioned fork().

I tried to explain the difference between registers and variables.

I’m not trying to show off or bring anyone down… I just like to help people. I’m old (my first Linux kernel commit was in 2004). And I could be wrong — please LMK if I made a factual error (I’d appreciate it, honestly).

All good?

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#202
post #96

Earlier quoted context omitted.

I have a different perspective: the underlying problem is calling setenv(). As far as I'm concerned, the environment is a read-only input parameter set on process creation like argv. It's not a mechanism for exchanging information within a process, as used here with SSL_CERT_FILE. And remember that the exec* family of calls has a version with an envp argument, which is what should be used if a child process is to be…

Indeed, environment variables should be used to configure child processes, not to configure the current process, for non-shell programs, IMHO. Note that Java, and the JVM, doesn't allow changing environment variables. It was the right choice, even if painful at times.

Java doesn't even allow to change the working directory also due to potential multi-threading problems.

Another reason why Java isn't the greatest language to create CLI tools with.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#203
post #14

Earlier quoted context omitted.

It's time to move beyond this attitude and make things safe by default. For example, Solaris has a safer version of setenv(). "It is ridiculous that this has been a known problem for so long. It has wasted thousands of hours of people's time, either debugging the problems, or debating what to do about it. We know how to fix the problem." https://www.evanjones.ca/setenv-is-not-thread-safe.html

One of the major differences between X Window and the win32 GUI APIs is that the windows one builds in thread safety, and it cannot be removed. This means that you pay the price of mutexes and the like (what the windows world likes to call "critical sections"), even if you have a single threaded GUI. X Window, on the other hand, decided to do nothing about threads at all, leaving it up to the application. 30 years af…

Meanwhile, macOS will check if the application is single threaded and if so avoid using locks: https://developer.apple.com/documentation/foundation/nsthrea...

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#204
post #85

This reminded me of that whole "12-factor app" movement, which several of my former coworkers had really bought into. One of the "factors" is that apps should be configured by environment variables. I always thought this was kinda foolish: your configuration method is a flat-namespace basked of stringly-typed values. The perils of getenv()/setenv()/environ are also, I think, a great argument against using env vars fo…

There probably should be an addendum to the "12-factor app" movement that says that the environment should be treated as read only for the duration of the process. Most of the issues people talk about here seem to relate to people trying to abuse the environment as some kind of key value store for mutable global state (which sounds like a bad idea). Why would you even want to do that?!

Being on the JVM which actually treats the environment as immutable that and which probably inspired a lot of the 12 factor app movement (with companies like Soundcloud being big Scala and Java users and pushing this), I've never experienced any issues with the environment changing on me or causing any threading issues. The environment is effectively immutable and there's nothing in my processes that sneakily circumvents that (via some native calls into libc). So, complete non issue on the JVM.

Even if somebody manages to modify the environment, the immutable copy stays the same. That copy gets created on JVM startup and is immutable. Anything using normal Java apis to interact with the environment will never see the modification. I'm sure people might have tried to work around that but it's not a wide spread practice. Because, again, why would you even want to do that?

The problem with configuration files is that their parsing is process specific. That's why Linux/Unix is such a mess. Every single tool seems to have its own conventions and mechanisms for configuration. There are no standards for this.

Other of course than the Docker ecosystem. You can do whatever you want inside the container but effectively your only interface to the outside world is either messily mounting some volume and doing whatever convoluted way of configuration your app requires; or just using environment variables. Most modern software is docker ready/friendly in the sense that you can fully control their behavior via the environment. It's perfectly adequate for most things that people run via docker these days. Which of course is pretty much anything.

And of course with Docker compose or kubernetes (which I'm not necessarily a fan of) you get yaml files defining lists of environment variables that define how your process starts. So you more or less get what you are asking for. I'm not a big YAML fan but it works well enough. Too much potential for syntax issues really ruining your day IMHO. But it's not like the alternatives are free of issues.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#205

Earlier quoted context omitted.

The problem with `setenv` is that people expect one process to have one set of environment variables, which is shared across multiple languages running in that process. This implies every language must let its environment variables be managed by a central language-independent library -- and on POSIX systems, that's libc. So if libc refuses to provide thread-safety, that impacts not just C, but all possible languages…

It's not just that "libc refuses to provide thread-safety" ... the POSIX standard specifies that these functions are non-reentrant.

A conformant implementation can make a non-reentrant function actually safe under the hood for people that call into it erroneously. Unfortunately, there is no way to do this for getenv/setenv, because of the API they expose (specifically, when environ is accessed directly).

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#206
post #151

Earlier quoted context omitted.

Mutating argv is fine for how it is usually done. That is, to permute the arguments in a getopt() call so that all nonoptions are at the end. It is fine because it is usually done during the initialization phase, before starting any other thread. setenv() can be used here too, though I prefer to avoid doing that in any case. I also prefer not to touch argv, but since that's how GNU getopt() works, I just go with it.…

A big reason to mutate argv is to change the process's name for tools like top.

For that you write to /proc/self/comm, that's where top gets it from.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#207
Click bait title? GLibC is very clear about what is and what is not thread-safe. I looked at the article: They fell victim to the classic getenv()/setenv() trap. This has been blogged about many times. If you look at the man page for setenv():

Ref: https://man7.org/linux/man-pages/man3/setenv.3.html

... it clearly says: "MT-Unsafe"

Also, there is a whole section about get/set env thread safety here (under "Other safety remarks -> env"):

https://man7.org/linux/man-pages/man7/attributes.7.html

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#208

Earlier quoted context omitted.

No amount of locking can make the getenv API thread-safe, because it returns a pointer which gets invalidated by setenv, but lacks a way to release ownership over it and unblock setenv safely (or to free a returned copy). So setenv's existence makes getenv inherently unsafe unless you can ensure the entire application is at a safe point to use them.

This is actually not that hard to fix. Getenv() could keep several copies of the value around: one internal copy protected by a mutex, that it never returns, and one copy per thread that it stores in thread local storage. When you call getenv(), it locks the mutex, checks if the current thread's value exists, populates it from the internal copy if not, and returns it. It will also install a new setenv-specific signal…

There has to be some sort of nuance regarding why this seemingly simple fix hasn't been made yet. Changing from crashing to blocking doesn't seem like a big breaking change.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#209
post #78

Its like a rite of passage to be hit by an environment related bug on linux, which is mysteriously less a problem on other unix's. Which is sorta funny given how pragmatic Linus and the kernel are about fixing POSIX bugs by making them not happen, while glibc is still lagging here decades after people tried to at least make the problem better. Sure there is all the crap around TZ/etc, but simply providing getenv_r()…

> environment related bug on linux, which is mysteriously less a problem on other unix's. How do you figure? The problem isn't the implementation, it's the API. setenv(), unsetenv(), putenv(), and especially environ, are inherently unsafe in a multithreaded program. Even getenv_r() can't really save you, since another thread may be calling setenv() while the (old) value of an env var is being copied into the provided…

>> environment related bug on linux, which is mysteriously less a problem on other unix's.

> How do you figure?

From https://illumos.org/man/3C/putenv:

> The putenv() function can be safely called from multithreaded programs

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#210
post #201

Earlier quoted context omitted.

No, it does not. I don't think you understand what you are talking about, because none of these actually address the points I brought up. They use the same words, but semantically they are talking about something completely different.

I provided a copy/paste from the site about the envp array size you asked about. I clarified why I mentioned fork(). I tried to explain the difference between registers and variables. I’m not trying to show off or bring anyone down… I just like to help people. I’m old (my first Linux kernel commit was in 2004). And I could be wrong — please LMK if I made a factual error (I’d appreciate it, honestly). All good?

I am going to do this once, but not again. Please pay attention to it. You are not just wrong, but failing to demonstrate an understanding of the actual topic being discussed. I can't say whether you actually have it or not, but your responses do not demonstrate this. I have dealt with plenty of people on this site who say things that are factually incorrect, many of whom have argued with me when I do so. You are not doing that; rather you are not even understanding what I am saying.

The article specifically mentions that the authors consulted the disassembly to see what was in x20. I know it is a general purpose register. They know it is a general purpose register. This knowledge is completely irrelevant: they read the code, they matched it against the actual source, they can confirm that at the time of crash x20 contains what they said it contains. The compiler optimizations have already run. They can't change anything anymore. That you mentioned this shows that you do not follow the actual order of events here.

envp, similarly, is in the process of being operated on in the crashing code. The authors grabbed its size from some random context at the time of the crash. The fact that it is not actually stored in the array itself is completely irrelevant to the fact that its numeric value was present in the crash dump. Obviously, some code that operated on it had computed the value and stashed it, which is a completely natural and expected thing for this code to do.

Finally, nobody cares about setenv across processes. The article didn't talk about this. It's completely irrelevant to mention this, and in fact there is another comment further down (which you may not have read, I'm ok with that) that also has the same confusion and it belies a poor grasp of what the actual problem is.

You can see that I am forced to do significantly more work than you to respond to what specifically is the problem here. It looks like you are pattern matching on specific words and then regurgitating your knowledge on it, whether it is relevant or not. When it's not, it's essentially just spam; when it is you fail to actually take into account the content that is actually being discussed. When I'm talking about how I almost got run over by a driver on their phone you are not welcome to step in and start talking about how a lot of hit-and-runs involve drunk drivers. I wasn't talking about a hit-and-run, and I just told you the person was on their phone. Somehow you completely missed that and kept talking about what you wanted to mention, like if you gave the gist of the conversation to someone else and asked them for their response on it and then pasted that here without checking to see if it was relevant or not. Don't do that.

Post reply on HN