Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

C stdlib isn't threadsafe and even safe Rust didn't save us

edgedb.com

Post reply on HN
Print view View original

221–230 of 370 posts

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#221

Post by fch42 »

GoblinSlayer wrote: post #196
lamontcg wrote: post #159

Earlier quoted context omitted.

Environment variables are a gigantic, decades-old hack that nobody should be using... but instead everyone has rejected file-based configuration management and everyone is abusing environment variables to inject config into "immutable" docker containers...

Everything already supports environment variables, and everyone and their dog have their own favorite yaml-based configuration management.

That (managing the env "from the outside) is and always has been the "supposed" way of using it.

Modifying _your own_ environment _at runtime_ is not. The corresponding functions - setenv/getenv - and state - envp/environ - have in the UNIX standards "always" (since threads exist, really) been marked non-MT. "way back when" people were happy to accept that stated restrictions on use don't make bugs. Today, general sense of overentitlement makes (some) people say "but since whatever-trickery can remove this restriction... you're wrong and I'm entitled to my bugfix". I agree the damage is done, though.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#222

Post by xxs »

diroussel wrote: post #107
eqvinox wrote: post #96

Earlier quoted context omitted.

I have a different perspective: the underlying problem is calling setenv(). As far as I'm concerned, the environment is a read-only input parameter set on process creation like argv. It's not a mechanism for exchanging information within a process, as used here with SSL_CERT_FILE. And remember that the exec* family of calls has a version with an envp argument, which is what should be used if a child process is to be…

Indeed, environment variables should be used to configure child processes, not to configure the current process, for non-shell programs, IMHO. Note that Java, and the JVM, doesn't allow changing environment variables. It was the right choice, even if painful at times.

>Note that Java, and the JVM, doesn't allow changing environment variables. It was the right choice, even if painful at times.

Not sure why would it be considered painful. Imo, use of setenv to modify your own variable, the definition of setenv is thread unsafe. So unless running a single threaded application it'd never make sense to call it.

Java does support running child processes with a designated env space (ProcessBuilder.environment is a modifiable map, copied from the current process), so inability to modify its own doesn't matter.

Personally I have never needed to change env variables. I consider them the same as the command line parameters.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#223

Post by xxs »

throwaway2037 wrote: post #212
Calzifer wrote: post #202

Earlier quoted context omitted.

Java doesn't even allow to change the working directory also due to potential multi-threading problems. Another reason why Java isn't the greatest language to create CLI tools with.

It is interesting that they do not allow ability to change env and working dir via security policy or a command line arg (--allow-setenv, etc.).

That would be so much wasted engineering effort. The actual solution is simple: read what you need from env, and pass it as parameters to the functions you want to. The values of what you have read can be changed... and if you really, really want start a child process with a modified env.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#224

Post by xxs »

debugnik wrote: post #132
usefulcat wrote: post #77

Earlier quoted context omitted.

But that won't actually fix the underlying problem, namely that getenv and setenv (or unsetenv, probably) cannot safely be called from different threads. It seems like the only reliable way to fix this is to change these functions so that they exclusively acquire a mutex.

No amount of locking can make the getenv API thread-safe, because it returns a pointer which gets invalidated by setenv, but lacks a way to release ownership over it and unblock setenv safely (or to free a returned copy). So setenv's existence makes getenv inherently unsafe unless you can ensure the entire application is at a safe point to use them.

They can have copy on write of course.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#225

Post by swiftcoder »

vrtx0 wrote: post #220
swiftcoder wrote: post #213

Earlier quoted context omitted.

> I hope this helps! It does not help, because you do not appear to have understood the article (or even read it all that closely). Some of these bullet points feel a lot like the kind of junk output one sees from the various (popular, but flawed) AI summary tools...

So, I’m real, and just trying to offer constructive feedback for a few errors I believe I noticed. I could be wrong though —- could you be specific? I don’t want to misinform anyone…

For 1 & 2, the issue wasn't that the author was looking at the wrong logs/coredumps. It's that coredumps from inside containers typically don't match the symbols available outside the container - you either have to run gdb inside a matching container, or rebuild the contents of the container in the host environment (as they did here).

3. There's nothing wrong with the level of abstraction here. If you have a crash that occurs on ARM but not on amd64, the differences in how those architectures operate is a very reasonable initial assumption.

4. The value in x20 is the same value in the local variable in question. Even though there may not be a general one-to-one mapping between variables and registers, at this particular instant in time that variable does correspond to this register.

5 is irrelevant, as the article isn't discussing forking. It's discussing the (somewhat questionable) practice of a program using getenv/setenv as mutable state.

For 6, the article doesn't say that env stores its own array length. It says that setenv called something like free() on the old env array, and free() overwrote env with the length of the memory allocation (which is a quite reasonable way for malloc to do book keeping).

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#226

Post by fch42 »

eqvinox wrote: post #119
gpm wrote: post #115

Earlier quoted context omitted.

> but argv is a function parameter to main()¹, and only that. > ¹ or technically whatever your ELF entry point is, _start in crt0 or your poison of choice. Once you include the footnote, at least on linux/macos (not sure about Windows), you could take the same perspective with regards to envp and the auxiliary array. It's libc that decided to store a pointer to these before calling your `main`, not the abi. At the ti…

I mean, yes, we're in "violent agreement" there. It's nice that libc squirrels away a copy and gives you a `getenv()` function with a string lookup, but… setenv… that was just a horrible idea. It's not really wrong to view it as a tool that allows you to muck around with main()'s local variables. Which to me sounds like one should take a shower after using it ;D (Ed.: the man page should say "you are required to take…

+1 on the "horrible idea" part.

Thing is, the (history of the) UNIX APIs - call'em "libc" if you like - is littered with the undead corpses of horrible ideas. Who thought that having global file write offsets are great ? Append-only writes ? Global working directories ? The ability to write the password db via putpwent() ? Modifying your own envp or argv ? Why have a horribly-scaling hack like fcntl-based file locking even in the standard ?

"Today", were one to start from scratch, the userspace API of even unix-ish operating systems would be done much differently. After all, systems designers and implementors are intelligent people and learn, and there's 50y+ of history to learn from. But the warts are there, and sometimes, there to "program around" them.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#227

Post by sim7c00 »

viraptor wrote: post #185
sim7c00 wrote: post #183

Earlier quoted context omitted.

just because someone makes a window doesn't mean you gotta jump out of it. there are good and bad uses for things, and the bad ones should be avoided lest one hurt themselves?

https://en.wikipedia.org/wiki/Death_of_Garry_Hoy people will assume more safety than necessary. You don't have to jump, but someone will try. We can accept that fact or watch people fail over and over on the same issue. It's better to help everyone avoid the problem in the first place. For some reason lots of programmers will behave like the comment section on an accident video. "I would notice that earlier", "I'd av…

doesn't make it less true. all sorts of 'dangerous' things are used by people daily all around the world. There are issues sure, but that doesn't mean it's neccesarily a bad thing. for example cars, stairs, kitchen knives...

These things are perhaps more commonly known to be bad, and the dangers are perhaps more obvious.

There will always be people who use things in the wrong way too, which doesn't make the thing bad, but how it's used.

There are buildings in my country with nets around them because people keep jumping off them (suicidal). The buildings are safe. The nets are not a solution, they just shift the problem and don't tackle the root cause.

There are many car crashes with fatal victims. Sure care manufacturers try to make cars safer, but there's no hordes of people hating on cars calling for them to be abolished in favor of safer technology because people rely on them heavily.

Same for libc. People try to improve its safety, and try to advice and write about its dangers. Just because bugs exist and unsafe conditions can occur doesn't mean something should be dropped all together... a lot of the world relies heavily on libc, safe and unsafe uses of it even.

What's more is that libc and linux etc. are open-source. If someone knows a sound solution to these issues which does not break the entire world, they are free to submit pull requests....

simply stating something is 'rubbish' and needs to be put down is an unproductive and shortsighted sentiment.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#228

Post by mark_undoio »

mmastrac wrote: post #46
Animats wrote: post #43

Earlier quoted context omitted.

Yes. That's known. Most of the rest of the problem here seems to be the development environment. They're testing on a remote machine in an Amazon data center and using Docker. This rig fails to report that a process has crashed. Then they don't have enough debug symbol info inside their container to get a backtrace. If they'd gotten a clean backtrace reported on the first failure, this would have been obvious. Why is…

Yup, it's mostly just the story and tools we used to get ourselves out of a mess that was made harder by some decisions made earlier -- the tests were running in a container with stripped symbols (we're going to ship symbols after this, no reason to over-optimize), our custom test runner failed to report process death (an oversight). There's no reason setenv should have been called here. The `openssl-probe` library c…

> Yup, it's mostly just the story and tools we used to get ourselves out of a mess that was made harder by some decisions made earlier -- the tests were running in a container with stripped symbols (we're going to ship symbols after this, no reason to over-optimize)

It's worth noting here that you can also build your binaries and keep debug symbols separately.

You don't need to ship them with the binary (although it will make many scenarios a bit simpler if you do, since you'll always have the right ones available).

Some info that might help: https://www.tweag.io/blog/2023-11-23-debug-fission/ https://undo.io/resources/gdb-watchpoint/reduce-binary-size-...

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#229

Post by fch42 »

josefx wrote: post #143
debugnik wrote: post #132

Earlier quoted context omitted.

No amount of locking can make the getenv API thread-safe, because it returns a pointer which gets invalidated by setenv, but lacks a way to release ownership over it and unblock setenv safely (or to free a returned copy). So setenv's existence makes getenv inherently unsafe unless you can ensure the entire application is at a safe point to use them.

C could provide functions to lock/unlock a mutex and require that any attempt to access the environment has to be done holding the mutex. This would still leave the correctness in the hands of the user, but at least it would provide a standard API to secure the environment in a multi threaded application that library and application developers could adopt.

That is basically "what it means" if an interface is non-MT: you can call this no-problem if you know you're singlethreaded, and if you're not, find your own way to serialize (meaning: have your own locking prinitive you acquire/release where you make calls to these functions).

One could "dream of" a func that tells libc "acquire/drop this mutex of mine around get/set/putenv calls" but that'd simply move the problem - because the nifty "frameworks" would do that (independently of each other, we're sovereign and entitled frameworks around here) and race each other's state nonetheless.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#230

Post by goeiedaggoeie »

usefulcat wrote: post #77
mmastrac wrote: post #5

The major takeaway from this is that Rust will be making environment setters unsafe in the next edition. With luck, this will filter down into crates that trigger these crashes ( https://github.com/alexcrichton/openssl-probe/issues/30 filed upstream in the meantime).

But that won't actually fix the underlying problem, namely that getenv and setenv (or unsetenv, probably) cannot safely be called from different threads. It seems like the only reliable way to fix this is to change these functions so that they exclusively acquire a mutex.

setenv and getenv have never been thread safe, why the concern with it now?
Post reply on HN