Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A “living” Linux process with no memory

github.com

Post reply on HN
Print view View original

31–40 of 117 posts

Re: Show HN: A “living” Linux process with no memory

#31

Post by Dylan16807 »

kick wrote: post #10
Wowfunhappy wrote: post #7

Earlier quoted context omitted.

I'd be much more reluctant to get on OP’s case about it than GP, but there’s a strong argument to be made that this hurts the overall python ecosystem.

There's also a strong argument to be made that people shouldn't have to care about ecosystems that are so delicate that a proof of concept in a previous version "hurts the overall [ecosystem]." "How dare you use a hammer! We're all using Hammer 2 now! I don't care that it was 'Only one nail," put it down and pick up Hammer 2! Your hammer is what, ten years old? That's ancient! It's not even compatible with my Hammer…

To make the analogy work I have to imagine that the two hammers cause horrible problems when used at the same time, and that they're both free with 5 minute shipping.

It's a sucky situation to be in, but we should probably get everyone on the same hammer.

(If we want a somewhat less nonsensical analogy, replace hammer with screwdriver. Two incompatible screw head patterns and you can only use one driver on any particular object.)

So sure don't nitpick a toy project, but if someone's writing significant code in python 2... it's not ideal.

-

But all of this is a dumb tangent, and we should just appreciate how funny it is to ask that this nonsensical pointless quasi-OS-breaking code be done in python 3 instead.

Re: Show HN: A “living” Linux process with no memory

#32

Post by jerf »

btown wrote: post #17
geofft wrote: post #11

I've now seen a similar case multiple times in the wild - if a process has a thread in uninterruptible sleep (e.g., blocked on a bad disk or a stuck network filesystem) and you kill it, the process dies, but the kernel waits forever for that thread before informing the parent process. The parent doesn't get SIGCHLD, and wait() doesn't return. (So, for example, your favorite init/supervisor won't restart the process o…

Can/do supervisors poll for this situation? Seems like that should be possible.

This becomes a tarpit. Whatever metric you think to measure for this may itself become a source of false positives in the field. Whatever things you choose to try to avoid that becomes another source of falseness in the system. This tends to cascade as people apply patch upon patch until you have a system that basically randomly fails and nobody can tell you why. This is not a theoretical report; in theory this approach can work, if you get it perfect enough. This is a practical report, both of systems I've built myself and of many other such cases I've witnessed, including some things around process management and supervision.

In the end, you really have little choice but to trust what the OS is telling you, and if the OS is lying for whatever reason, to fix the OS. As hard as that last bit may be as a solution, it still turns out to be the easiest, most-effective fix. And it may not be a perfect fix, either... but it'll still be the best.

Re: Show HN: A “living” Linux process with no memory

#33

Post by kick »

Dylan16807 wrote: post #31
kick wrote: post #10

Earlier quoted context omitted.

There's also a strong argument to be made that people shouldn't have to care about ecosystems that are so delicate that a proof of concept in a previous version "hurts the overall [ecosystem]." "How dare you use a hammer! We're all using Hammer 2 now! I don't care that it was 'Only one nail," put it down and pick up Hammer 2! Your hammer is what, ten years old? That's ancient! It's not even compatible with my Hammer…

To make the analogy work I have to imagine that the two hammers cause horrible problems when used at the same time, and that they're both free with 5 minute shipping. It's a sucky situation to be in, but we should probably get everyone on the same hammer. (If we want a somewhat less nonsensical analogy, replace hammer with screwdriver. Two incompatible screw head patterns and you can only use one driver on any partic…

No "horrible problems" arise from Python 2 being used on the same system that Python 3 is on. The only problems are trivial, which was highlighted in the analogy.

Re: Show HN: A “living” Linux process with no memory

#34

Post by kristjansson »

kees99 wrote: post #28

Did anybody try to run it? Python part of this PoC depends on python2 module "fuse", which in turn depends on "gunpowder", "a library to facilitate machine learning on large, multi-dimensional images". What is even going on here? $ pip2 install fuse Collecting fuse Downloading https://files.pythonhosted.org/packages/c3/f6/82777531d0dd0fa1d1b509258873f4b48e1ec702dcf0258214fafb474895/fuse-0.1.3.tar.gz ERROR: Packages i…

I think you want https://pypi.org/project/fuse-python/ instead. That python packages can and often do have different names than their corresponding distributables is ... not great.

Re: Show HN: A “living” Linux process with no memory

#35

Post by cryptonector »

xenadu02 wrote: post #25
btown wrote: post #17

Earlier quoted context omitted.

Can/do supervisors poll for this situation? Seems like that should be possible.

Also curious why the kernel can't abandon the thread since the process is dead anyway; set a bit in kernel space so on the walk back from the syscall it bails out instead of trying to return to userspace. That doesn't solve the kernel thread being stuck forever problem. I'm not sure what the fix is there... is it bad drivers with no timeout mechanism? I don't know how they'd do that unless the kernel IO is all async…

The kernel can't abandon that thread entirely, but it could "detach" it from the process it was part of and arrange for it to clean itself up and disappear if ever it wakes. And so the parent process could get told of the child's death.

Re: Show HN: A “living” Linux process with no memory

#36

Post by phkahler »

btown wrote: post #17
geofft wrote: post #11

I've now seen a similar case multiple times in the wild - if a process has a thread in uninterruptible sleep (e.g., blocked on a bad disk or a stuck network filesystem) and you kill it, the process dies, but the kernel waits forever for that thread before informing the parent process. The parent doesn't get SIGCHLD, and wait() doesn't return. (So, for example, your favorite init/supervisor won't restart the process o…

Can/do supervisors poll for this situation? Seems like that should be possible.

>> Can/do supervisors poll for this situation?

Sounds like a hack.

Re: Show HN: A “living” Linux process with no memory

#37

Post by Dylan16807 »

state_less wrote: post #19
stevemk14ebr wrote: post #12

Earlier quoted context omitted.

we're programmers, anything can be done with enough time or money

Can you write a program to tell if any given program is going to halt?

As long as linear execution time is acceptable, okay. Give me a couple million dollars and I'll have it to you in a week.

Re: Show HN: A “living” Linux process with no memory

#38

Post by Dylan16807 »

kick wrote: post #33
Dylan16807 wrote: post #31

Earlier quoted context omitted.

To make the analogy work I have to imagine that the two hammers cause horrible problems when used at the same time, and that they're both free with 5 minute shipping. It's a sucky situation to be in, but we should probably get everyone on the same hammer. (If we want a somewhat less nonsensical analogy, replace hammer with screwdriver. Two incompatible screw head patterns and you can only use one driver on any partic…

No "horrible problems" arise from Python 2 being used on the same system that Python 3 is on. The only problems are trivial, which was highlighted in the analogy.

I didn't say same system, I said same time.

As in, trying to run one program with both.

Re: Show HN: A “living” Linux process with no memory

#39

Post by Wowfunhappy »

kick wrote: post #33
Dylan16807 wrote: post #31

Earlier quoted context omitted.

To make the analogy work I have to imagine that the two hammers cause horrible problems when used at the same time, and that they're both free with 5 minute shipping. It's a sucky situation to be in, but we should probably get everyone on the same hammer. (If we want a somewhat less nonsensical analogy, replace hammer with screwdriver. Two incompatible screw head patterns and you can only use one driver on any partic…

No "horrible problems" arise from Python 2 being used on the same system that Python 3 is on. The only problems are trivial, which was highlighted in the analogy.

I wrote this reply earlier and deleted it. Part of me feels like it's too far afield, but it's also kind of not, so I'll include it. It's what's on the top of my mind right now:

---

I'm a volunteer teacher at a Girls Who Code club. We use Python. I spent the last class before the universe collapsed into Cornoavirus hell introducing the kids to the concept of functions.

After introducing the concept, I asked the kids to modify a quiz program they'd written with a different teacher to use functions to ask a question and check the answer, instead of copying and pasting the code each time. This was a pretty challenging assignment—these kids hadn't encountered functions before—but one of the girls managed to get it and I was really proud of her.

Except her code wouldn't run because—I realized with horror—the quiz program she'd written with another teacher was actually python 2 code!

I am still, weeks later, so damn frustrated that this happened. I'm trying to get the kids to wrap their heads around functions; I don't want to start explaining the political BS around python 2 and 3.

It's primarily my fault, of course, and secondarily the python foundation's fault for handling this transition so badly. But every new project that's done in python 2—whether serious or a toy—is also contributing to the issue in a very small way.

So while this isn't a conversation I would have personally brought up in this thread, I think it's all just very unfortunate. Python is a nice language and it doesn't deserve this crap.

Re: Show HN: A “living” Linux process with no memory

#40

Post by jerf »

hsnewman wrote: post #8

Can this be implemented in Go?

I think so, but probably not reliably and/or for the long term.

In terms of the actual unmap call, the payload is in the munmap call, which is a syscall, and if you do have to set up the registers for that call, Go does have its assembler support which should be able to do that for you. You should be able to write Go source code that would perform that call, although you're going to be poking into some corners of Go most programmers never have to get into. You can also get to fuse, which in my experience is great at producing uninterruptible sleeps even when you're not trying to do it on purpose....

However, in terms of reliability and the long term, you have the problem that you're not the only thread in the program the way a C program can count on that, and if any of those other threads wake up and their memory is missing, it'll probably result in the process dying. You can do a couple things to try to avoid that a little, like pinning yourself to an OS thread and running the program in one CPU mode, but especially on the latest version of Go where they've implemented true pre-emption, you can't keep the runtime from running by just hogging the execution thread anymore.

So I suspect that as long as you win the race to unmap everything before the runtime wakes up (which may also require you to call sysmap through the assembly code directly, rather than using the syscall libraries as I'm pretty sure those notify the scheduler of what you're doing), you may be able to briefly be a process with no RAM mapped, but in human terms it won't be long before the runtime wakes up to do something, anything, and crash the process as a result. There's no way to avoid that in Go. You won't be able to have a process just sitting there indefinitely with no mapped RAM that you can admire and treasure and hand off to your children as part of their inheritance.

Post reply on HN