Linux Namespaces and Go Don't Mix • Hacker News Board

Re: Linux Namespaces and Go Don't Mix

#131

Post by cyphar » Sat, Jun 03, 2017, 7:02 AM UTC

lunixbochs wrote: post #46

vbernat wrote: post #18

Earlier quoted context omitted.

In Linux, many properties are task-related and therefore the API is also task-related. Namespaces are not the only problem. See for example https://github.com/golang/go/issues/1435 . The problem is Go runtime does not provide a way to exclusively lock a system thread from existing or future routines. Most other languages would work just fine (C or Python for example).

> The problem is Go runtime does not provide a way to exclusively lock a system thread from existing or future routines. runtime.LockOSThread() does exactly that [1]. [1] https://golang.org/pkg/runtime/#LockOSThread

The documentation for both GOMAXPROCS and runtime.LockOSThread lie to you. Neither of them allow you to stop the runtime from creating new threads (or executing library code in the wrong thread).

Trust me, we've been trying to get Go to co-operate with containers for quite a few years. It's not as simple as just reading the standard library docs. ;)

Re: Linux Namespaces and Go Don't Mix

#132

Post by cyphar » Sat, Jun 03, 2017, 7:04 AM UTC

liveoneggs wrote: post #37

this is just a more complex version of the issue where go can't safely do the daemonize dance for a privileged port

And that's unfortunately only an issue because Linux doesn't implement POSIX's model for thread privileges. In particular, glibc has to implement the nptl in userspace to make Linux threads look like POSIX threads.

I get the feeling that the Go runtime doesn't do the same.

Re: Linux Namespaces and Go Don't Mix

#133

Post by cyphar » Sat, Jun 03, 2017, 7:08 AM UTC

YZF wrote: post #124

the8472 wrote: post #118

Earlier quoted context omitted.

But linux threads predate go, so go is built on assumptions they knew were not true.

It's possible they didn't know different Linux threads can be in different namespaces, it's not exactly something everyone knows. Namespaces in Linux aren't that new (15 years or so). It's possible they knew but decided this wasn't an important enough use case to warrant language features. I've never needed to have half my process in one namespace and half in another. It's a niche application. Green threads/goroutine…

> Someone should file a Go bug and see what the response is...

From memory, several people working on Docker have done so over the years. It's still a problem because the only "real" solutions are:

1. Do what glibc does and implement nptl(7) (effectively a way to make Linux threads look like POSIX threads by synchronising certain operations on all threads). This would require making first-class library APIs for Linux features (outside of the wild-west that is syscall).

2. Give programs far more control over threading, which would require making runtime.LockOSThread and GOMAXPROCS actually do what their documentation says. However, that would restrict their ability to be opinionated about threading (and would almost certainly cause deadlocks in some programs) so I understand why they don't want to do this either.

Re: Linux Namespaces and Go Don't Mix

#134

Post by cyphar » Sat, Jun 03, 2017, 8:40 AM UTC

klodolph wrote: post #74

jankedeen wrote: post #70

Earlier quoted context omitted.

No. You can call fork pretty cleanly in a pthread based program. This post is not informed criticism. There is some prep for ugly contingencies(pthread_atfork) but it usually just works. Namespace based code while doing the same is like setting your hair on fire while running through the gasoline forest..but caveat emptor is usually spelled out pretty critically in the API and docs. Oh, btw: Fuck RUST. For once and f…

> There is some prep for ugly contingencies(pthread_atfork) but it usually just works. Actually, that's not true. Perhaps it's theoretically possible, in some cases, to make your threaded code work with fork with the appropriate pthread_atfork() handlers, but in general, it's a total mess. Thread A has a lock, thread B calls fork(), thread B tries to acquire the lock. The lock is held, but thread A is not running in…

On paper, all code between a fork and exec (in a multithreaded process) must be async-signal safe as well. I'm not sure if this is also true for the callbacks with pthread_atfork (I've personally never used it), but it's something to keep in mind.

Re: Linux Namespaces and Go Don't Mix

#135

Post by bhawks » Sat, Jun 03, 2017, 8:44 AM UTC

I think the proper title is: Linux Process and Threads Don't Mix.

The Linux syscall interface exposes certain functionalities that are much more easy to reason about at the process level such as namespaces, capabilities, seteuid and so on. However these syscalls all operate on the thread level (since the kernel treats threads pretty similarly to processes). Therefore in order to perform these operations safely you need some sort of process wide mechanism to apply the operation on every thread (and don't forget error handling!)

This is _not_ just a golang problem or an M:N threading problem as many comments suggest. The kernel really needs to provide new syscalls for these features that operate at the process / thread-group level. The current syscalls are extremely difficult to use correctly in any multithreaded context in any language. When you consider the security implications of these features it makes the problem even worse.

Check out https://ewontfix.com/17/ for a really good analysis of the difficulty musl libc has faced making a multi-thread safe seteuid on Linux. There are also many bugs in glibc related to this as well. Linux makes userspace responsible for patching up the leaks in the kernel's process abstraction and that's really not a job that userspace is in the right position to take on.

Re: Linux Namespaces and Go Don't Mix

#136

Post by cyphar » Sat, Jun 03, 2017, 8:47 AM UTC

jankedeen wrote: post #81

steveklabnik wrote: post #77

Earlier quoted context omitted.

> Oh, btw: Fuck RUST. For once and for all fuck this tyranny of coercion by potential IP holders. It is a shit language. ... what? I am confused as to what you're saying here.

* Coercion: large down vote/propaganda community against any negative rust idea..evidenced here. * IP Holders, well this is speculative. I can't understand anyone who would compare Rust against C in a positive way without possibly profiting from it. Rust is painful to write, painful to learn and makes you ask yourself: Can I not write a well designed and correct program in C? Of course I can. Why Am I Learning Rust?…

> Rust is painful to write, painful to learn and makes you ask yourself:

It's painful to write because writing software that is memory safe is not easy. Rust makes it insanely easy in comparison to the "old way" of doing it (hacking together a C program then having to patch security holes every release, never being certain that your code is actually memory safe).

> Can I not write a well designed and correct program in C? Of course I can.

Of course you can. All it takes is a separate formal proofing process, countless hours of static analysis and many more months of fixing security bugs. Good thing that's so much simpler than Rust's static analysis which runs on every compilation and mathematically proves that its memory model was not violated.

I think if we have learned one thing in the history of C programs, it's that this sentiment that "a sufficiently smart programmer can write safe C code" is quite harmful. While technically true, I don't think I've ever met such an individual and I doubt one exists.

Re: Linux Namespaces and Go Don't Mix

#137

Post by mappu » Sat, Jun 03, 2017, 9:28 AM UTC

cyphar wrote: post #130

mappu wrote: post #122

Earlier quoted context omitted.

> You can't use cgo to control the threading of the Go runtime itself, which is the real problem here. Go is deliberately opinionated about threading of the runtime. I think it's unlikely Go will offer much more control over these internals (beyond GOMAXPROCS), given the philosophy around e.g. GC tuning. Cgo is a beefed-up runtime.LockOSThread() that could be used to avoid having the author resort to a helper process…

> I think it's unlikely Go will offer much more control over these internals (beyond GOMAXPROCS) GOMAXPROCS is actually a lie. If you set it to 1, the runtime will still create threads.

> GOMAXPROCS is actually a lie. If you set it to 1, the runtime will still create threads.

It's not a lie. The variable is GOMAXPROCS, not GOMAXTHREADS. "procs" is specific jargon of the G:M:P scheduler design.

Re: Linux Namespaces and Go Don't Mix

#138

Post by laumars » Sat, Jun 03, 2017, 9:45 AM UTC

catern wrote: post #101

concede_pluto wrote: post #99

Earlier quoted context omitted.

> Can I not write a well designed and correct program in C? We have decades of coredumps and exploits to convince us that nobody can. How many well-known apps can you name that have never blown up randomly? I can't think of a single one. How much more failure until it's reasonable to think that C requires an inhuman level of perfection and almost any alternative would be an improvement? I hear good things about seL4…

GNU ls has never segfaulted or otherwise blown up for me. :)

GNU Bash segfaults for me whenever I type ~[tab]. Thankfully this only happens on the small few systems I had to manually patch after the epic Bash vulnerability a couple of years ago - older internal systems that are still in use for historic reasons rather than regularly relied upon in production. So it's never been annoying enough to fix.

Re: Linux Namespaces and Go Don't Mix

#139

Post by the8472 » Sat, Jun 03, 2017, 9:47 AM UTC

bhawks wrote: post #135

I think the proper title is: Linux Process and Threads Don't Mix. The Linux syscall interface exposes certain functionalities that are much more easy to reason about at the process level such as namespaces, capabilities, seteuid and so on. However these syscalls all operate on the thread level (since the kernel treats threads pretty similarly to processes). Therefore in order to perform these operations safely you ne…

> The kernel really needs to provide new syscalls for these features that operate at the process / thread-group level.

Or it could provide another clone flag that indicates that threads spawned that way should share privileges and similar things, then runtimes that need threads to all behave the same way can opt into that. I suspect that some tools do advanced privilege kung-fu that relies on those per-thread properties.

Re: Linux Namespaces and Go Don't Mix

#140

Post by cyphar » Sat, Jun 03, 2017, 11:33 AM UTC

mappu wrote: post #137

cyphar wrote: post #130

Earlier quoted context omitted.

> I think it's unlikely Go will offer much more control over these internals (beyond GOMAXPROCS) GOMAXPROCS is actually a lie. If you set it to 1, the runtime will still create threads.

> GOMAXPROCS is actually a lie. If you set it to 1, the runtime will still create threads. It's not a lie. The variable is GOMAXPROCS, not GOMAXTHREADS. "procs" is specific jargon of the G:M:P scheduler design.

Yeah, you're correct. Though it doesn't change that it's a deceptively named variable -- you have to read the runtime docs carefully to realise that it only refers to the number of threads concurrently executing Go code.