Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I summarized my understanding of Linux systems

github.com

Post reply on HN
Print view View original

71–80 of 88 posts

Re: I summarized my understanding of Linux systems

#71

Post by [deleted] »

Jasper_ wrote: post #48
timeforcomputer wrote: post #21

Nice! I want to do something similar and map my understanding of Linux. I find some diagrams on Wikipedia fascinating (example: https://en.m.wikipedia.org/wiki/File:Linux_Graphics_Stack_20... , but more to do with the user library ecosystem rather than kernel and program runtime). These diagrams make me want to learn about each part and be able to comprehend in principle what is happening on my machine. Eventually...

Any diagram by ScotXW on Wikipedia is somewhere between misleading and completely wrong, and they're a constant pain on the Linux graphics community. If you're curious about the details in this case, ScotXW confuses EGL and OpenGL, the arrows aren't quite right, and the labels aren't quite right either (DRM is labeled "hardware-specific" but KMS isn't? The label for "hardware specific Userspace interface to hardware…

[deleted]

Re: I summarized my understanding of Linux systems

#72

Post by deaddodo »

temeya wrote: post #37
guerrilla wrote: post #31

Earlier quoted context omitted.

This is one of my favorite books. A true classic. There are follow-ups in that style for Linux and FreeBSD as well. I think Robert Love wrote the former.

And Marshall Kirk McKusick wrote the latter, "The Design and Implementation of the FreeBSD Operating System"

I used this book as a primary reference for OS design (along with The Dinosaur Book) when designing my hobby OS.

The FreeBSD kernel/world is almost exquisite in it's engineering simplicity. Compared to the sometimes chaotic world of GNU/Linux (in my experience).

Re: I summarized my understanding of Linux systems

#73

Post by deaddodo »

ooterness wrote: post #38
Koshkin wrote: post #23

Earlier quoted context omitted.

> execute assembly I always thought that assembly was a type of programming language.

Ben Eater's excellent educational video series includes one explaining the difference: https://www.youtube.com/watch?v=oO8_2JJV0B4 In short, assembly for a given CPU is very nearly one-to-one with the machine language for that CPU. It's not correct to conflate the two, but close enough when speaking informally.

While it's true that assembler doesn't quite fit into the same class as compiled/interpreted languages, it would be disingenuous to say it's not a programming language. It's simply a very low-level, machine specific one.

It's even blurrier when you consider that most modern assembler dialects have plenty of high level functionality (structs, macros, labels, etc) that do not correlate to machine instructions.

Re: I summarized my understanding of Linux systems

#74

Post by persnickety »

icedchai wrote: post #49
persnickety wrote: post #13

Earlier quoted context omitted.

If you're using a hypervisor, then the userspace program inside the VM is also using the CPU and memory directly. You'd have to do full emulation to avoid that. Even with full emulation, I'd say memory is being accessed directly, unless you really go out of your way to make it weird.

With full emulation, I'd argue memory access is not direct. Memory access from the emulated system will go through user space code in the emulator. That code may translate it to actual memory access, or perhaps an emulated, memory mapped I/O device like a frame buffer. Either way, there is something in the middle. You could argue that nothing is direct unless you're running on a bare metal system, no MMU, no page tab…

For the definition I used earlier, the bytes don't change shape in between. That makes MMU and interpreted access direct, and compression indirect.

Re: I summarized my understanding of Linux systems

#75

Post by suprjami »

akira2501 wrote: post #68
suprjami wrote: post #67

Earlier quoted context omitted.

Hardware ring has nothing to do with "directly using the CPU", it controls what access level the program has. Forget virtualisation. Compile a userspace program which just adds numbers into a stack variable. That program is running directly on the CPU in the unprivileged ring. A userspace program in a VT-x virtual machine is exactly the same. If those programs attempt privileged access then that access will fail and…

> Hardware ring has nothing to do with "directly using the CPU" Why wouldn't it? Several features are simply not available in ring 3. Several features are configured for you in a way you cannot change. Several instructions will simply fault your program. > which just adds numbers into a stack variable Yes.. and when you eventually overflow that stack, what happens? How did the stack segment selector get created? Can…

You don't have adminstrative access to all of Hacker News. Therefore, you are not really on Hacker News. This is your logic.

Re: I summarized my understanding of Linux systems

#76

Post by arghwhat »

candiodari wrote: post #52
arghwhat wrote: post #36

Earlier quoted context omitted.

Having to know structs is not really an issue - you also need to know text formats, JSON schemas, what not. Re-ordering of structs is always forbidden with the binary format being strictly specified, so there's nothing to worry about there. Can't exactly shuffle bytes in a text format either, and plan9 control strings tend to have positional arguments. The current structs do leave something to be desired though.

Really? The issue is that those structs don't cross-compile correctly if you aren't very careful in their use. I've personally managed to write an IRC bot that had a select loop that worked fine on i386 but didn't on amd64 until I figured out what was happening to the size of the structure behind it.

The kernel decided to have a different uAPI on different architectures. This is not just struct size that might differ, but the fields available and their possible values. Imagine a JSON field being a numeric constant in one arch, but a string in another. I get why they did it, but it certainly causes... surprises.

I imagine the issue you experienced might have been either that you hardcoded the numeric size instead of using sizeof and then building on a different architecture, or that you redeclared the struct for one architecture, ignoring that you were building for another architecture.

Not to undermine the annoyance or surprise-factor of your bug, I am tempted to say that it falls into generic logic bugs rather than the fault of any system. Changing to an encoded with dynamic length for example would just introduce different classes of logic bugs, like the whole series of possible issues within JSON parsing/serialization and requiring dynamic buffers.

A struct on the other hand only requires referencing the right struct declaration - the only amount of care you need is to include and use them. In my opinion, this is the maximum possible convenience for such system.

Re: I summarized my understanding of Linux systems

#77

Post by vbezhenar »

sophacles wrote: post #55
wrs wrote: post #45

Earlier quoted context omitted.

I think the point of the diagram is that the abstractions or “API” that user space gets to use includes memory it can read and write directly, and a CPU to execute instructions. Of course in reality there only “appears to be” memory and a CPU, but that’s why it’s an abstraction. Just like there “appears to be” a filesystem for user space to use, when in reality there’s a block interface to a disk, or wherever you wan…

I think what I was getting at was that memory sort of sits in-between. My instructions are executed directly on the cpu. My file reads and writes directly translated from a stream of bytes to block instructions by code in the kernel. Memory is a wierd in-between place, or maybe a 3rd option, since the kernel has to run a bunch of code on my behalf for me to use memory, sort of like the filesystem thing, but I'm using…

CPU is pre-configured by OS as well. So IMO it's the same as memory. May be it would be more appropriate to say that userspace program accesses directly some parts of CPU and RAM.

With modern computers, AFAIK even OS does not have absolutely full control over CPU.

Re: I summarized my understanding of Linux systems

#78

Post by sophacles »

vbezhenar wrote: post #77
sophacles wrote: post #55

Earlier quoted context omitted.

I think what I was getting at was that memory sort of sits in-between. My instructions are executed directly on the cpu. My file reads and writes directly translated from a stream of bytes to block instructions by code in the kernel. Memory is a wierd in-between place, or maybe a 3rd option, since the kernel has to run a bunch of code on my behalf for me to use memory, sort of like the filesystem thing, but I'm using…

CPU is pre-configured by OS as well. So IMO it's the same as memory. May be it would be more appropriate to say that userspace program accesses directly some parts of CPU and RAM. With modern computers, AFAIK even OS does not have absolutely full control over CPU.

But they don't directly access memory. They access an address that may be translatable by the mmu to a physical location in ram. They may also write to an address that the kernel hasn't allocated a page for yet, but that the kernel has agreed to map into the process' memory. In this case the kernel handles the trap and maps a page of actual ram (etc) and then the process continues forward progress.

Re: I summarized my understanding of Linux systems

#79

Post by justsomehnguy »

vbezhenar wrote: post #77
sophacles wrote: post #55

Earlier quoted context omitted.

I think what I was getting at was that memory sort of sits in-between. My instructions are executed directly on the cpu. My file reads and writes directly translated from a stream of bytes to block instructions by code in the kernel. Memory is a wierd in-between place, or maybe a 3rd option, since the kernel has to run a bunch of code on my behalf for me to use memory, sort of like the filesystem thing, but I'm using…

CPU is pre-configured by OS as well. So IMO it's the same as memory. May be it would be more appropriate to say that userspace program accesses directly some parts of CPU and RAM. With modern computers, AFAIK even OS does not have absolutely full control over CPU.

> With modern computers, AFAIK even OS does not have absolutely full control over CPU.

It's even more amusing with Type 2 hypervisors.

Re: I summarized my understanding of Linux systems

#80

Post by zozbot234 »

arghwhat wrote: post #15
vbezhenar wrote: post #12

Earlier quoted context omitted.

Everything is a descriptor. When I'm opening a TCP connection, there's no file, so calling it a file descriptor feels wrong. And at that point, the whole "everything is" turns into nonsense, because yes, everything is a pointer to something, so what.

There are named files (which have a file path) and anonymous files (which do not). You can see these in /proc/$PID/fd/$FD if you're curious - when the link doesn't start with '/', it's anonymous. Even process memory is just an anonymous file on Linux, and arguably a cleaner one as it operates on proper fds, instead of plan9 where a string "class name" (not a path) is used to access the magical '#g' filesystem. The di…

> Their design is pretty much solely a limitation caused by the idea that all filesystems should exist through the 9p protocol, which as a networked protocol cannot share data (fds, structs), only copy (payloads to/from read, write). With the idea that all functionality had to be replaceable and mountable from remote machines, the only possible API became read/write for everything.

It's not clear to me that 9p itself could not be extended to allow for shared memory. With low-level control over the operating system and rebuilding of existing binaries, distributed shared memory becomes a possibility. (I.e. the existing VM system ought to be enough to implement whatever cache coherence is needed for shared memory over the network.)

Post reply on HN