Live data from Hacker News

I summarized my understanding of Linux systems

github.com

61–70 of 88 posts

Re: I summarized my understanding of Linux systems

#61
post #57

Earlier quoted context omitted.

Having to know structs is not really an issue - you also need to know text formats, JSON schemas, what not. Re-ordering of structs is always forbidden with the binary format being strictly specified, so there's nothing to worry about there. Can't exactly shuffle bytes in a text format either, and plan9 control strings tend to have positional arguments. The current structs do leave something to be desired though.

Structured json data that you mentioned does allow to “shuffle bytes” It doesn’t matter which order the client sends {“a”: 1, “b”: 2 }, it’s one object with “a” and “b” regardless

Do the same to a JSON array.

Position having meaning is a common part of most data representations, not something specific to structs. If you need to, you can also engineer order independence in structs with arrays of type and value unions, but there's no need to as the order has never been a problem.

Re: I summarized my understanding of Linux systems

#62

A future simple linux-like (or unix-like) OS -- could theoretically be created with only 4 syscalls : open() read() write() close() Such a theoretical linux-like or unix-like OS would assume quite literally that "everything is a file" -- including the ability to perform all other syscall/API calls/functions via special system files , probably located in /proc and/or /sys and/or other special directories, as other pos…

You've effectively reinvented 9p here. Which is good!

There are some differences which may interest you: https://9fans.github.io/plan9port/man/man9/intro.html

I think you may find that some of the additional complexity of 9p is necessary, but perhaps not all of it.

Re: I summarized my understanding of Linux systems

#63
post #33

Whenever I've made notes like this, it's never been useful to my future self nor to anyone else. The only use I've had of this kind of documentation is that the process of writing it, made me understand it better. Basically write-only documentation. I would call myself a Linux expert, and while I can kinda see what you mean with this diagram, it would not have been useful to me back before I was an expert.

I found it useful. Allowed me to compare if I have similar idea to the author.

Re: I summarized my understanding of Linux systems

#64

A future simple linux-like (or unix-like) OS -- could theoretically be created with only 4 syscalls : open() read() write() close() Such a theoretical linux-like or unix-like OS would assume quite literally that "everything is a file" -- including the ability to perform all other syscall/API calls/functions via special system files , probably located in /proc and/or /sys and/or other special directories, as other pos…

I had considered that too, but what I had also considered, and that I think is better, is a different single syscall, which is more like a actor model or like a capability-based system. (One problem with the "everything is a file" like Plan9 is that then the operating system has to parse the file paths every time you want to do any I/O; what I describe below ignores that problem since you can link directly to objects instead.)

A process has access to a set of capabilities (if it does not have any capabilities, then it is automatically terminated (unless a debugger is attached), since there is nothing for the program to do).

A "message" consists of a sequence of bytes and/or capabilities. (The message format will be system-independent (e.g. the endianness is always the same) so that it works with emulation and network transparency, described below.)

A process can send messages to capabilities it has access to, receive messages from capabilities it has access to, create new capabilities (called "proxy capabilities"), discard capabilities, and wait for capabilities.

Terminating the process is equivalent to a mandatory blocking wait for an empty set of capabilities; discarding all capabilities also terminates the process. A non-blocking wait for an empty set of capabilities means that you wish to yield, so that other processes get a chance to run, before this process continues.

Some further options may be needed to handle multiple locking and transactions, and to avoid some kinds of race conditions, but mostly that is just it.

This is useful for many things, including sandboxing, emulation, network transparency (this can be done by one program keeping track of which capabilities need to be sent across the network link and assign an index number to each one, and then the other end will create a proxy capability for each index number and use that number when it wants to send back), security with user accouts, etc; the kernel does not need to know about all of these things since they can be implemented in user code.

Other things (outside of the kernel) can also be implemented in terms of proxy capabilities, and I had ideas about those other parts of the operating system too, for example it has a hypertext file system (with no file names, but files can contain multiple numbered streams, which can include both bytes and links to other files (which can be either to the current version or to a fixed version; if to a fixed version then copy-on-write will be used if the file is modified)), and the "foreign links table", and a common (binar) data format, and a command shell with some similarities than Nushell (but also many differences), and the system uses the "Extended TRON Code" character set, and details about the working of the package manager and IME and window manager, etc.

Re: I summarized my understanding of Linux systems

#65
post #33

Whenever I've made notes like this, it's never been useful to my future self nor to anyone else. The only use I've had of this kind of documentation is that the process of writing it, made me understand it better. Basically write-only documentation. I would call myself a Linux expert, and while I can kinda see what you mean with this diagram, it would not have been useful to me back before I was an expert.

Usually I would agree. I typically make a "Crash-Course in $PLATFORM" document while keeping notes. These I very commonly reference in order to externalize my memory since it seems to be approaching capacity. I don't care about Ruby on Rails, but once I did, and I can reference my notes if I ever need to touch that platform again.

Re: I summarized my understanding of Linux systems

#66
post #23

Earlier quoted context omitted.

Nor does mine. Userspace assembly runs directly on the CPU* executing in the unprivileged ring. When the userspace program makes a system call by calling a kernel entry function which is mapped into the process's address space by the dynamic loader, then part of that entry into kernelspace is to put the CPU into the privileged ring and kernel assembly then runs on the CPU. The process scheduler can stop execution to…

> execute assembly I always thought that assembly was a type of programming language.

Here I'm referring to assembly as mnemonic for machine code, but yes it would have been more correct to say machine code instead.

Re: I summarized my understanding of Linux systems

#67

Earlier quoted context omitted.

Userspace program directly uses CPU and memory (unless you're using VM). In contrast to that, your userspace program does not directly access your network device or SSD, but uses kernel routines to access those indirectly.

Ring 3 is not "directly using the CPU." And mmap is not "directly using the memory."

Hardware ring has nothing to do with "directly using the CPU", it controls what access level the program has.

Forget virtualisation. Compile a userspace program which just adds numbers into a stack variable. That program is running directly on the CPU in the unprivileged ring.

A userspace program in a VT-x virtual machine is exactly the same.

If those programs attempt privileged access then that access will fail and a trap is raised. That's what the CPU ring controls.

Re: I summarized my understanding of Linux systems

#68

Earlier quoted context omitted.

Ring 3 is not "directly using the CPU." And mmap is not "directly using the memory."

Hardware ring has nothing to do with "directly using the CPU", it controls what access level the program has. Forget virtualisation. Compile a userspace program which just adds numbers into a stack variable. That program is running directly on the CPU in the unprivileged ring. A userspace program in a VT-x virtual machine is exactly the same. If those programs attempt privileged access then that access will fail and…

> Hardware ring has nothing to do with "directly using the CPU"

Why wouldn't it? Several features are simply not available in ring 3. Several features are configured for you in a way you cannot change. Several instructions will simply fault your program.

> which just adds numbers into a stack variable

Yes.. and when you eventually overflow that stack, what happens? How did the stack segment selector get created? Can you change that selector or it's attributes? Can you set the stack pointer to any valid memory address you like?

> A userspace program in a VT-x virtual machine is exactly the same.

What does an IOMMU do?

> If those programs attempt privileged access then that access will fail and a trap is raised.

Right.. so you are not directly using the CPU. You're not even in control of what timeslices are afforded to you by the OS. You are in an exceptionally limited environment most of which you cannot control or alter and much of which you cannot even observe.

The fact that instructions get dispatched according to the system ABI when you run a program is not material to this problem, and in particular, is not at all correctly represented by this diagram.

Re: I summarized my understanding of Linux systems

#69

Earlier quoted context omitted.

Everything is a descriptor. When I'm opening a TCP connection, there's no file, so calling it a file descriptor feels wrong. And at that point, the whole "everything is" turns into nonsense, because yes, everything is a pointer to something, so what.

There are named files (which have a file path) and anonymous files (which do not). You can see these in /proc/$PID/fd/$FD if you're curious - when the link doesn't start with '/', it's anonymous. Even process memory is just an anonymous file on Linux, and arguably a cleaner one as it operates on proper fds, instead of plan9 where a string "class name" (not a path) is used to access the magical '#g' filesystem. The di…

> magical '#g' filesystem.

Whats magical about the segmet(3)[1] device? The '#' devices are kernel file servers. There's no magic.

[1] http://man.9front.org/3/segment

Re: I summarized my understanding of Linux systems

#70

Earlier quoted context omitted.

Hardware ring has nothing to do with "directly using the CPU", it controls what access level the program has. Forget virtualisation. Compile a userspace program which just adds numbers into a stack variable. That program is running directly on the CPU in the unprivileged ring. A userspace program in a VT-x virtual machine is exactly the same. If those programs attempt privileged access then that access will fail and…

> Hardware ring has nothing to do with "directly using the CPU" Why wouldn't it? Several features are simply not available in ring 3. Several features are configured for you in a way you cannot change. Several instructions will simply fault your program. > which just adds numbers into a stack variable Yes.. and when you eventually overflow that stack, what happens? How did the stack segment selector get created? Can…

You are directly using the CPU, you just do not have full access to the entire CPU. There is no userspace ALU that your numbers get crunched on, there is no userspace register file your working set is stored in (actually, they might do that internally, but logically there is no such distinction). You are in a hotel room. Just because you can not stomp around in the ducts does not mean you are not directly using the hotel room, you just have limited access to the rest of the hotel.
Post reply on HN