141–150 of 184 posts
On macOS, fork() is a bit weird: https://opensource.apple.com/source/Libc/Libc-997.90.3/sys/f... Many frameworks are backed by XPC services, where the parent process has a socket-like connection to a backend server. After forking, the child would have no valid connection to the server. The fork() function establishes a new connection in the child for libSystem, to allow Unix programs to port easily to macOS, but othe…
Earlier quoted context omitted.
It's not about being "uncharitable". It's about protecting nix from being controlled by outside forces. MS does, indeed. have world-class research, but they are sticking their heads in the nix camp, which some of us don't like. We're not all in this together, despite what some will tell you. Sadly, UNIX (umbrella term here) is not what it was a few years ago. I dearly miss Solaris, for example. Nothing touched it in…
Microsoft Research is not Microsoft. Microsoft Research employs some of the main Haskell developers, and you don't see Haskellers going all conspiracy theory. Research is research, and either the ideas they describe are good and should be adopted, or they're bad and should be ignored.
Earlier quoted context omitted.
Cleaning up your own process between fork and exec is hard. Several programs resort to terrible hacks like force-closing everything except file IDs 0,1,2 in a loop. Or they look into their /proc directory to discover whichnfile IDs exist, which is only marginally better. But when your process is a house of cards built on third party libraries with their own minds, there are not a lot of other options.
Use O_CLOEXEC everywhere (even third party libs). It's really annoying, but necessary. Means you need to use accept4(), dup3(), popen with an additional "e" (of course all of that needs to be feature tested, during compilation/runtime).
When I learnt how fork() and select() worked, I just felt in love with Unix. The Win32 API was so ad-hoc and unnatural in direct comparison.
For me it was poll() due to it's simple and intuitive API. Also, it's much faster then select() when you have a large number of file descriptors being monitored.
Earlier quoted context omitted.
That's true, it's not the only reason. Dealing with threads and buffers and pthread_atfork and the associated heartbreak is a biggie also. But the performance is nothing to laugh at. I just did a quick test, a 100mb process generally takes >2ms to fork, while a 1mb or less process takes 70us. It seems like its pretty much linear with process size.
Does using hugepages help?
Earlier quoted context omitted.
About shared libraries, I know that there is this line of thought considering them "evil" (well at least sufficiently to want to get rid of them); but I'm quite unsure about what a modern system would look like without them (although this is less a problem at the application level on e.g. Android, the system level is still extremely important) With Spectre, proper process bounds (well, address spaces) are more import…
We are using Linux instead of Hurd due to manpower. Most high integrity real time OSes are microkernels. Interesting that you mention Android, one of the key points of Project Treble is using separate processes for drivers with Android IPC to talk to the kernel (including hardware buffer handles).
So let's put manpower kind of aside for the bulk of the dev (where thousands of man-years are needed for any big project) and look at what could actually be achieved with the very small amount of manpower bootstrapping those projects. At this point you understand that the manpower thing is only a convenient narrative, while the reality is that even at the early time, Linux based systems worked really better than Hurd based systems.
Because general purpose micro-kernels based systems are hard, and especially those with a design as ambitious as the Hurd. (When you start to want to strongly isolate FS from VM code, it even stops being just hard and starts to be really HARD.)
And this was even worse at the time for perf reasons (but perf reasons are still applicable even today, given the impact on mobile and datacenter workloads)
However, yes, I'm in favor of more isolation today, because for a shitload ton of drivers it literally won't make any difference whether or not you take 1us vs 50us if you need to execute once every few seconds. So it is retarded to the highest level to run in kernel space if you don't actually need it. Sadly, Linux is way behind on that subject today.
That being said, and back to the original subject, a microkernel or at least a less monolithic one won't really get us in the less shared-libraries direction if it just re-implements the same perimeter of features of a monolithic ones, nor in the huge pages everywhere direction...
Earlier quoted context omitted.
The last time I used WSL (perhaps 6 months ago), its per-process overhead was awful. I don't recall the numbers, but I think it managed to start fewer than 10 processes per second. My memory suggests it was more like two processes per second, though I would recommend re-testing before trusting that. Found my previous comment on it (which has a test case but not numbers): https://news.ycombinator.com/item?id=18226921
On 1803 and 1903 it's 3 to 4 times faster than MsysGit (WSL is ~1s on my laptops). It is possibly slightly faster on 1903 as my laptop running it is faster than the other for this bench, despite having an older processor. Now in a Linux VM it's approx 10 times faster than even WSL. And that should probably be even faster natively. So anyway WSL is really usable and if you really only started 10 processes per sec some…
Earlier quoted context omitted.
On 1803 and 1903 it's 3 to 4 times faster than MsysGit (WSL is ~1s on my laptops). It is possibly slightly faster on 1903 as my laptop running it is faster than the other for this bench, despite having an older processor. Now in a Linux VM it's approx 10 times faster than even WSL. And that should probably be even faster natively. So anyway WSL is really usable and if you really only started 10 processes per sec some…
Well, I hadn't installed any antiviruses myself. I think Windows Defender was running, though. It's possible that my computer came with additional crapware on it.
When I disable it, it is down to ~0.5s
I would not build a Linux kernel here instead of in a VM, but for tons of things, this is very usable.
While fork() might be sub-optimal for launching different programs (fork() + exec() vs. posix_spawn()), it's absolutely essential in several types of common systems that don't use it to launch different programs. Fork-requiring program class 1: The biggest example where fork() is needed are webservers/long-running programs with significant unchanging memory overhead and/or startup time. Many large applications writte…
Another common use of fork() for things other than exec()ing is multi-process services where all will keep running te same program. Arranging to spawn or vfork-then-exec self and have the child realize it's a worker and not a (re)starter is more work because a bunch of state needs to be passed to the child somehow (via an internal interface), and that feels hackish... And also this case doesn't suffer much from fork()s badness: you fork() early and have little or no state in the parent that could have fork-unsafety issues. But it's worth switching this use-case to spawn or vfork-then-exec just so we have no use cases for fork() left.