Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I summarized my understanding of Linux systems

github.com

Post reply on HN
Print view View original

41–50 of 88 posts

Re: I summarized my understanding of Linux systems

#41

Post by moody__ »

arghwhat wrote: post #15
vbezhenar wrote: post #12

Earlier quoted context omitted.

Everything is a descriptor. When I'm opening a TCP connection, there's no file, so calling it a file descriptor feels wrong. And at that point, the whole "everything is" turns into nonsense, because yes, everything is a pointer to something, so what.

There are named files (which have a file path) and anonymous files (which do not). You can see these in /proc/$PID/fd/$FD if you're curious - when the link doesn't start with '/', it's anonymous. Even process memory is just an anonymous file on Linux, and arguably a cleaner one as it operates on proper fds, instead of plan9 where a string "class name" (not a path) is used to access the magical '#g' filesystem. The di…

I am not exactly sure where you got the "class name" part from, I've typically refereed to those as kernel filesystems or sharp devices. For the record these kernel filesystems are not technically 9p, they present an interface much similar but reads and writes to them are not marshaled and unmarshaled from 9p. It is however possible to export their files over 9p if one desires, I can import a remote machines /net stack and use it to announce or dial out. Plan 9 gives us proto-VPNs just be virtue of its design.

There was perhaps a time where the differences between having everything in binary ioctls and bound to specifically one device was a necessary component in order to reach reasonable performance, but I don't believe that is the case anymore. Anecdotally these days everything on Plan 9 feels snappier. We have some benchmarks that show that 9front outperforms linux with naive pipe io and context switches. What Plan 9 misses in micro optimizations it makes up for by having a incredibly consistent and versatile base.

I want to reiterate the benefits of the network transparency by talking about how drawterm works. Drawterm can be thought of the plan 9 equivalent of windows RDP. How it works is that internally drawterm creates routines to expose a /dev/draw, /dev/mouse and /dev/keyboard through whichever native way there is on the target system (macos, windows, linux, etc). It then attaches to the remote system and overlays these files over a namespace. Programs like our window manager rio can then be run completely transparently, forwarding not compressed images, but individual draw RPC messages. There is no need for any special code on the plan 9 host side in order to accommodate drawterm, again it is something that just falls out of the core design of the system.

Re: I summarized my understanding of Linux systems

#42

Post by moody__ »

Cloudef wrote: post #18
arghwhat wrote: post #17

Earlier quoted context omitted.

Now, proper plan9-style namespaces, that I miss. :) User namespaces are still a hell of a lot clunkier than "each process inherits its parents' namespace".

If you setup user namespace, the child processes will inherit that namespace. The difference is that plan9 is fully built on this idea and isn't multi-user, on linux you have to opt-in to this. It's very useful and underused (mostly used by containers). I wanted to ship my AWS lambdas this way, but sadly AWS lambdas don't allow user namespaces. https://github.com/aws/aws-lambda-base-images/issues/143

Plan 9 is very multi-user, namespaces are actually one component in how it is multi user specifically. A given terminal may be designed to only have one user on it but CPU servers are still multi-user multiplexers, that part has not been given up.

Re: I summarized my understanding of Linux systems

#43

Post by Cloudef »

moody__ wrote: post #41
arghwhat wrote: post #15

Earlier quoted context omitted.

There are named files (which have a file path) and anonymous files (which do not). You can see these in /proc/$PID/fd/$FD if you're curious - when the link doesn't start with '/', it's anonymous. Even process memory is just an anonymous file on Linux, and arguably a cleaner one as it operates on proper fds, instead of plan9 where a string "class name" (not a path) is used to access the magical '#g' filesystem. The di…

I am not exactly sure where you got the "class name" part from, I've typically refereed to those as kernel filesystems or sharp devices. For the record these kernel filesystems are not technically 9p, they present an interface much similar but reads and writes to them are not marshaled and unmarshaled from 9p. It is however possible to export their files over 9p if one desires, I can import a remote machines /net sta…

Even on linux people avoid syscalls because syscalls are slow and bad. So I don't really see the problem with plan9's approach either. Make the common scenario useful, optimize for special cases (sendfile, io_uring). In fact read/write lets you batch bigger amount of data than single ioctl can actually be more performant.

Re: I summarized my understanding of Linux systems

#44

Post by Cloudef »

moody__ wrote: post #42
Cloudef wrote: post #18

Earlier quoted context omitted.

If you setup user namespace, the child processes will inherit that namespace. The difference is that plan9 is fully built on this idea and isn't multi-user, on linux you have to opt-in to this. It's very useful and underused (mostly used by containers). I wanted to ship my AWS lambdas this way, but sadly AWS lambdas don't allow user namespaces. https://github.com/aws/aws-lambda-base-images/issues/143

Plan 9 is very multi-user, namespaces are actually one component in how it is multi user specifically. A given terminal may be designed to only have one user on it but CPU servers are still multi-user multiplexers, that part has not been given up.

Yes, by multi-user i meant, plan9 doesn't have the unix/linux user model, but rather "multi-user" is done with the namespaces. Container people would be more familiar with the plan9 way.

Re: I summarized my understanding of Linux systems

#45

Post by wrs »

sophacles wrote: post #30
vbezhenar wrote: post #10

Earlier quoted context omitted.

Userspace program directly uses CPU and memory (unless you're using VM). In contrast to that, your userspace program does not directly access your network device or SSD, but uses kernel routines to access those indirectly.

It doesn't directly access memory. The addresses in your userspace program are not the actual addresses of memory in the ram sticks - there is a table of mappings that the kernel sets up. When your process asks the kernel for memory, it says "i need 5KB, please put that at address XYZ". The kerenel goes and finds 5KB unused, probably at some other address ABC, and creates a mapping in the table that says XYZ translat…

I think the point of the diagram is that the abstractions or “API” that user space gets to use includes memory it can read and write directly, and a CPU to execute instructions. Of course in reality there only “appears to be” memory and a CPU, but that’s why it’s an abstraction. Just like there “appears to be” a filesystem for user space to use, when in reality there’s a block interface to a disk, or wherever you want to draw the line.

Re: I summarized my understanding of Linux systems

#46

Post by projektfu »

richardwhiuk wrote: post #22

I don't understand what the boxes on this diagram are meant to represent. It feels like an elaborate mechanism to draw something wrong in the hopes people will correct it.

FWIW, I also don't really understand what the boxes are supposed to represent, given that the arrows represent dependencies like PID To me, a block diagram might show [CPU Scheduler], [Virtual Device Manager], [VFS Manager], [Memory Manager], [Interrupt Handlers], etc...

Of course, my knowledge of Linux internals is limited and perhaps it has a separation of the concept of PID and process where there is a literal dependency.

Re: I summarized my understanding of Linux systems

#47

Post by bigfatfrock »

guerrilla wrote: post #31
tmalsburg2 wrote: post #26

I learned a lot about this from the book "The design of the Unix operating system" by Maurice J. Bach.¹ It's an old book and many details deviate from actual present-day Linux, but it nonetheless gives a great overview of the key components and ideas. ¹ https://books.google.de/books/about/The_Design_of_the_UNIX_O...

This is one of my favorite books. A true classic. There are follow-ups in that style for Linux and FreeBSD as well. I think Robert Love wrote the former.

Thank you! I was going to ask for the latest linux variant - are you speaking of Love's "Linux Kernel Development", or "Linux in a Nutshell"?

I've been a primary linux user for a couple decades now but I'm not too keen on digging into kernel hacking but love details like the OPs post.

Re: I summarized my understanding of Linux systems

#48

Post by Jasper_ »

timeforcomputer wrote: post #21

Nice! I want to do something similar and map my understanding of Linux. I find some diagrams on Wikipedia fascinating (example: https://en.m.wikipedia.org/wiki/File:Linux_Graphics_Stack_20... , but more to do with the user library ecosystem rather than kernel and program runtime). These diagrams make me want to learn about each part and be able to comprehend in principle what is happening on my machine. Eventually...

Any diagram by ScotXW on Wikipedia is somewhere between misleading and completely wrong, and they're a constant pain on the Linux graphics community.

If you're curious about the details in this case, ScotXW confuses EGL and OpenGL, the arrows aren't quite right, and the labels aren't quite right either (DRM is labeled "hardware-specific" but KMS isn't? The label for "hardware specific Userspace interface to hardware specific direct rendering manager" is quite weird), and some important boxes are flat out missing. It's nitpicking for sure, but when the diagram goes out of its way to add extremely weird details, it demands nitpicking.

Nobody in the Linux graphics community would draw the diagram like this.

Re: I summarized my understanding of Linux systems

#49

Post by icedchai »

persnickety wrote: post #13
vbezhenar wrote: post #10

Earlier quoted context omitted.

Userspace program directly uses CPU and memory (unless you're using VM). In contrast to that, your userspace program does not directly access your network device or SSD, but uses kernel routines to access those indirectly.

If you're using a hypervisor, then the userspace program inside the VM is also using the CPU and memory directly. You'd have to do full emulation to avoid that. Even with full emulation, I'd say memory is being accessed directly, unless you really go out of your way to make it weird.

With full emulation, I'd argue memory access is not direct. Memory access from the emulated system will go through user space code in the emulator. That code may translate it to actual memory access, or perhaps an emulated, memory mapped I/O device like a frame buffer. Either way, there is something in the middle.

You could argue that nothing is direct unless you're running on a bare metal system, no MMU, no page tables. How do you define "direct"?

Re: I summarized my understanding of Linux systems

#50

Post by guerrilla »

bigfatfrock wrote: post #47
guerrilla wrote: post #31

Earlier quoted context omitted.

This is one of my favorite books. A true classic. There are follow-ups in that style for Linux and FreeBSD as well. I think Robert Love wrote the former.

Thank you! I was going to ask for the latest linux variant - are you speaking of Love's "Linux Kernel Development", or "Linux in a Nutshell"? I've been a primary linux user for a couple decades now but I'm not too keen on digging into kernel hacking but love details like the OPs post.

The Kernel Development book. It gives a tour. Read the UNIX one first though.
Post reply on HN