Can it be done without fuse?
Fuse is restricted; probably for reasons like this.
Show HN: A “living” Linux process with no memory
91–100 of 117 posts
Re: Show HN: A “living” Linux process with no memory
#92Did anybody try to run it? Python part of this PoC depends on python2 module "fuse", which in turn depends on "gunpowder", "a library to facilitate machine learning on large, multi-dimensional images". What is even going on here? $ pip2 install fuse Collecting fuse Downloading https://files.pythonhosted.org/packages/c3/f6/82777531d0dd0fa1d1b509258873f4b48e1ec702dcf0258214fafb474895/fuse-0.1.3.tar.gz ERROR: Packages i…
Use your OS's own packaging system to install python-fuse.
It may already be installed!
Re: Show HN: A “living” Linux process with no memory
#93I've now seen a similar case multiple times in the wild - if a process has a thread in uninterruptible sleep (e.g., blocked on a bad disk or a stuck network filesystem) and you kill it, the process dies, but the kernel waits forever for that thread before informing the parent process. The parent doesn't get SIGCHLD, and wait() doesn't return. (So, for example, your favorite init/supervisor won't restart the process o…
I saw my first lich on SPARCstation 10 (or 20?).
It was a java program that I killed -9. It died but was never reaped.
It prevented a software shutdown so I had to use the ol' STOP-A
Re: Show HN: A “living” Linux process with no memory
#94Can it be done without fuse? Fuse is restricted; probably for reasons like this.
Ha. It isn't. User namespaces allow you to do this attack:
I have a reproduction here:
https://github.com/sargun/fuse-example
Basically, what’s happening is that the FUSE daemon which is the one handling the FUSE requests has /dev/fuse open. There is a thread in that FUSE daemon. let’s call the FUSE daemon P10, and the thread P11.
P10 wires up a FUSE filesystem on /tmp, it opens /dev/fuse with FD5 P11 enacts an uninterruptible (blocking operation) on /tmp/foo, called OP1, and /tmp/foo is FD6. P10 reads the operation (OP1) from /dev/fuse, so now OP1 is in userspace ----The Pid 1 of the namespace is killed--
P10 just terminated, and cannot make progress. It will never respond to OP1. FD5 and FD6 remain open, because P11 is in uninterruptible sleep. P11 is in an uninterruptible disk sleep waiting for OP1 to respond, and the fuse connection never aborts.
FUSE abortion doesn’t kick in because the FD of `/dev/fuse` that P10 originally opened as the FUSE daemon under FD5 will not be closed until all threads as part of the process are terminated. The mount namespace wont be torn down until P11 is torn down. P11 will never be torn down because it’s waiting for someone to do something.
Re: Show HN: A “living” Linux process with no memory
#95As someone who doesn't understand much of what's going on here, what resources would you suggest I study to improve?
A more pragmatic book is The Linux Programming Interface, by Michael Kerrisk. It is Linux-specific though, but that's probably fine.
Uninterruptible process sleep is covered in section 22.3. Threads, in chapters 29–33. Signals, chapters 20–22. The proc file system in section 12.1. Memory mappings, chapters 49–50.
Re: Show HN: A “living” Linux process with no memory
#96Earlier quoted context omitted.
you could also mount nfs with soft timeouts which won't wait forever and instead return an error to userspace.
Which is a very good practice if you're in control of the software receiving the error. But so much freaking software doesn't handle this correctly it's maddening and so sleep everything that touches it is the only safe thing to do.
For some network filesystems, I want to mount them "Kill every process with a filehandle open to this on error". Is there a way to do that?
Re: Show HN: A “living” Linux process with no memory
#97Earlier quoted context omitted.
Can/do supervisors poll for this situation? Seems like that should be possible.
>> Can/do supervisors poll for this situation? Sounds like a hack.
Typically supervisors are monitoring readiness/liveness of the managed process for other reasons; you don't have the number of replicas you've promised if one of the replicas is un-live (perhaps hitting this bug, perhaps internally deadlocked, perhaps internally waiting forever on something the OS doesn't care about, like a certain network request), and you shouldn't be routing network traffic to a service that isn't ready for it.
So while I agree with the other posts that a supervisor can't recover from this state, it can be aware that it's happened. And it's quite common for supervisors to check for readiness and liveness beyond "well the OS says it's running, sounds good to me."
Re: Show HN: A “living” Linux process with no memory
#98Did anybody try to run it? Python part of this PoC depends on python2 module "fuse", which in turn depends on "gunpowder", "a library to facilitate machine learning on large, multi-dimensional images". What is even going on here? $ pip2 install fuse Collecting fuse Downloading https://files.pythonhosted.org/packages/c3/f6/82777531d0dd0fa1d1b509258873f4b48e1ec702dcf0258214fafb474895/fuse-0.1.3.tar.gz ERROR: Packages i…
Use your OS's own packaging system to install python-fuse. It may already be installed!
The python-fuse package on Ubuntu 18.04 does not seem to work (too old?).
$ python2 fs.py x
Traceback (most recent call last):
File "fs.py", line 8, in
class fs(fuse.Operations):
AttributeError: 'module' object has no attribute 'Operations'
$ python2
Python 2.7.17 (default, Nov 7 2019, 10:07:09)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import fuse
>>> fuse.__version__
'0.2.1Re: Show HN: A “living” Linux process with no memory
#99Earlier quoted context omitted.
Which is a very good practice if you're in control of the software receiving the error. But so much freaking software doesn't handle this correctly it's maddening and so sleep everything that touches it is the only safe thing to do.
For some network filesystems, I want to mount them "Kill every process with a filehandle open to this on error". Is there a way to do that?
NFS has been converted to to the TASK_KILLABLE mentioned elsewhere in the comments, so you could scan for threads stuck in this state, scan proc for open file descriptors pointing to the network filesystems and then kill those processes.
That of course doesn't work if you're dealing with a filesystem that still uses TASK_UNINTERRUPTIBLE
Re: Show HN: A “living” Linux process with no memory
#100Earlier quoted context omitted.
It kind off depends on what exactly you don't understand, but a book about Operating Systems and Linux Programming are probably what you want to read. They'll teach you about threads, segfault, the meaning of uninterruptible sleep, signals, the /proc/ directory, etc.
Can you suggest a book about Operating Systems?
I'd suggest "Operating Systems: Three Easy Pieces". (http://pages.cs.wisc.edu/~remzi/OSTEP/)
I've found it very accessible. Largely, the way that they teach is to describe a system based on a set of assumptions, then slowly relax each of the assumptions one by one until you reach an example of a real system. That style of teaching really works for me.