C stdlib isn't threadsafe and even safe Rust didn't save us • Hacker News Board

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#321

Post by skissane » Thu, Jan 23, 2025, 8:05 PM UTC

Calzifer wrote: post #202

diroussel wrote: post #107

Earlier quoted context omitted.

Indeed, environment variables should be used to configure child processes, not to configure the current process, for non-shell programs, IMHO. Note that Java, and the JVM, doesn't allow changing environment variables. It was the right choice, even if painful at times.

Java doesn't even allow to change the working directory also due to potential multi-threading problems. Another reason why Java isn't the greatest language to create CLI tools with.

> Java doesn't even allow to change the working directory also due to potential multi-threading problems.

Linux and macOS both support per-thread working directory, although sadly through incompatible APIs.

Also, AFAIK, the Linux API can't restore the link between the process CWD and thread CWD once broken – you can change your thread's CWD back to the process CWD, but that thread won't pick up any future changes to the process CWD. By contrast, macOS has an API call to restore that link.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#322

Post by kurante » Thu, Jan 23, 2025, 9:09 PM UTC

loeg wrote: post #299

colonial wrote: post #283

TIL that my set_env("RUST_LOG"...) calls at startup are technically unsafe. Funny. I should see if the env_logger crate has a better solution.

At startup it's probably fine! It's safe in a single-threaded environment.

As long as they don't use `#[tokio::main]` or any other attribute that wraps main into an async function!

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#323

Post by pjmlp » Thu, Jan 23, 2025, 9:29 PM UTC

danudey wrote: post #303

harrall wrote: post #301

Earlier quoted context omitted.

You can’t convince me that there is EVER a reason to call setenv() after program init as part of a regular program, outside needing to hack around something specific. Environmental variables are not a replacement for your config. It’s not a place to store your variables. Even if the env var API is fully concurrent, it is not convention to write code that expects an env var to change. There isn’t even a mechanism for…

> You can’t convince me that there is EVER a reason to call setenv() after program init as part of a regular program, outside needing to hack around something specific. The most common use I see for this is people setting an env in the current process before forking off a separate process; presumably because they don't realize that you can pass a new environment to new processes. I wonder what bugs you'd find if you…

Given how old most UNIX APIs are, and that when I do man fork I get information to look into execve(), which provices the feature, I guess not knowing is a typical case from google-copy-paste programming.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#324

Post by nyrikki » Thu, Jan 23, 2025, 9:32 PM UTC

acuozzo wrote: post #270

lamontcg wrote: post #159

Earlier quoted context omitted.

Environment variables are a gigantic, decades-old hack that nobody should be using... but instead everyone has rejected file-based configuration management and everyone is abusing environment variables to inject config into "immutable" docker containers...

> instead everyone has rejected file-based configuration management With good reason. Files are surprisingly hard: https://danluu.com/deconstruct-files/

Rejecting one hard problem and replacing it with another method that is officially documented to be worse isn't really a solution.

Note the standard:

https://pubs.opengroup.org/onlinepubs/009604499/functions/se...

> The setenv() function need not be reentrant. A function that is not required to be reentrant is not required to be thread-safe.

With the increased use of PIE, thunks for both security and due to ARM + the difference between glibc and musl, plus busybox and you have a huge mess.

I would encourage you to play around with ghidra, just to see what return oriented programming and ARM limits does.

Compilers have been good at hiding those changes from us, but the non-reentrant nature will cause you issues even without threads.

Hint, these thunks can get inserted in the MI lowering stage or in the linker.

But setenv() is owned by posix, with only getenv() being differed to cppr.

Perhaps someone could submit a proposal on how to make it reentrant to the Open Group. But it wasn't really intended for maintaining mutable state so it may be a hard sell.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#325

Post by pjmlp » Thu, Jan 23, 2025, 9:33 PM UTC

fch42 wrote: post #226

eqvinox wrote: post #119

Earlier quoted context omitted.

I mean, yes, we're in "violent agreement" there. It's nice that libc squirrels away a copy and gives you a `getenv()` function with a string lookup, but… setenv… that was just a horrible idea. It's not really wrong to view it as a tool that allows you to muck around with main()'s local variables. Which to me sounds like one should take a shower after using it ;D (Ed.: the man page should say "you are required to take…

+1 on the "horrible idea" part. Thing is, the (history of the) UNIX APIs - call'em "libc" if you like - is littered with the undead corpses of horrible ideas. Who thought that having global file write offsets are great ? Append-only writes ? Global working directories ? The ability to write the password db via putpwent() ? Modifying your own envp or argv ? Why have a horribly-scaling hack like fcntl-based file lockin…

You can see how that would look like, done by UNIX authors themselves, by looking into Inferno and Limbo standard library.

It is kind of ironic how so many stick with UNIX and C ideas as religious ideals from OS and systems programming ultimate design, while the authors moved on creating Plan 9 and Inferno, Alef and Limbo.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#326

Post by pjmlp » Thu, Jan 23, 2025, 9:38 PM UTC

loeg wrote: post #296

kibwen wrote: post #276

Earlier quoted context omitted.

The concern now is that, unlike when Posix was set in stone, threads exist.

The p in pthreads stands for Posix. I.e., uh, Posix is neither set in stone, nor entirely predates threads.

I am old enough to remeber when UNIX only had processes, and several thread designs were being discussed until eventually pthreads one design won.

POSIX predates adoption of threads in the UNIX world.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#327

Post by SAI_Peregrinus » Thu, Jan 23, 2025, 9:41 PM UTC

rerdavies wrote: post #120

jcotton42 wrote: post #95

Earlier quoted context omitted.

It's not just libc, it's any C or C++ library that calls getenv or setenv.

Specifically, any C or C++ library that calls setenv (despite documentation that says that setenv is not threadsafe).

Or any multithreaded program that uses a C or C++ library that calls setenv somewhere internally, and failed to document that it does so and is thus unsuitable for use by multithreaded programs.

No library does that documentation, so you can't use libraries on POSIX systems if writing multithreaded code. Or you do and hope for the best. So everyone just hopes for the best.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#328

Post by debugnik » Thu, Jan 23, 2025, 9:47 PM UTC

HarHarVeryFunny wrote: post #281

4gotunameagain wrote: post #181

Earlier quoted context omitted.

I think the argument was that the standard states that setenv is not thread safe, although from what I see it says that it does not have to be thread safe: The setenv( ) function need not be thread-safe. A function that is not required to be thread-safe is not required to be reentrant. https://www.open-std.org/jtc1/sc22/open/n4217.pdf . Page.. 1860 :')

Sure, but given that Linux defines the environment as state that's shared between threads, not having a thread-safe way of accessing it is hard to defend... Is "the standard says it doesn't NEED to be thread safe" the argument that the Linux libc maintainers are using for not enhancing it to be thread safe, or is it based on some technical or backwards compatibility issues in doing so ?

The only thread-safe way to implement getenv/setenv as they currently exist is to leak the previous state when setenv allocates, such that existing pointers stay valid. The existing API simply lacks a mechanism to synchronize correctly.

Leaking would be good enough for many use cases, but it would break long-running users of setenv (mainly those with libraries abusing env vars, as in TFA), and doesn't even solve how they interact with putenv and environ. This whole API is just cursed.

Libc could of course get better APIs, like GetEnvironmentVariable on Windows, but that won't fix all existing code.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#329

Post by api » Thu, Jan 23, 2025, 10:13 PM UTC

bluGill wrote: post #271

api wrote: post #258

Earlier quoted context omitted.

There are other ways that could be achieved, like cloud storage constantly mirroring local but encrypted with local keys or keys controlled by the user. This is the iCloud model and it works. Imagine a more open version with competing storage providers. This, however, would hand control back to the user, which would be bad for the software industry with its addiction to lock in and recurring revenue.

You also need the apps installed on whatever then, and enough CPU power to run those apps. I'm not saying you are wrong, but there is a lot of nuance here.

For CPU power, a Raspberry Pi today is faster than servers that ran whole medium to large businesses 20 years ago. Much of what people do with SaaS involves backend processes that could run on a 1990s era PC.

There are exceptions, like large AI models and huge databases like web search, though in the case of AI models I can run pretty decent ones locally already, but on an admittedly expensive laptop. If the rate at which models grow is not as fast or faster than the rate at which computers grow, mainstream PCs or even phones will catch up eventually.

I've actually wondered if that might be a major factor that swings the pendulum back... if you can run an AI that has memorized the entire Internet locally, that makes all kinds of things possible in local compute.

Installing apps could be easy, even automatic on demand. That's kind of what the web does. Imagine the web with better caching of program objects, maybe a runtime built around WASM, and an iCloud-type data model, and you can visualize personal computing for today. The kludgy idea of installers that vomit files all over the system is already legacy.

But it would still break SaaS lock-in, so this isn't where the money goes. Our software paradigms wrap themselves around whatever works as a business model.

Re: C stdlib isn't threadsafe and even safe Rust didn't save us

#330

Post by bashkiddie » Thu, Jan 23, 2025, 10:29 PM UTC

msully4321 wrote: post #11

01HNNWZ0MV43FF wrote: post #6

Earlier quoted context omitted.

Env vars are good if you treat them as read-only within the process

Yeah, setenv should probably just not exist, and environment variables should be only set when spawning new processes.

I guess the usecase is

    dlopen()

which passes on your environment. If you want to load libpam-keberos and pass

    DEBUG=verbose

you will need to setenv() your own environment.