Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple’s Darwin OS and XNU Kernel Deep Dive

tansanrao.com

Post reply on HN
Print view View original

11–20 of 186 posts

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#11

Post by p_ing »

kccqzy wrote: post #9
whalesalad wrote: post #3

I’ve been wanting to understand Darwin at this depth for a long time. Great read!

I've also wanted to understand Windows NT at this depth for a while. Skip the Win32 stuff, and discuss what's underneath it. As I understand Win32 is just one personality; there was also Windows Services for UNIX in the Windows XP days and Subsystem for UNIX-based Applications in Windows Vista. The underlying NT kernel is flexible enough to allow POSIX compliance. That would be an interesting read.

Windows Internals is the book you want.

Or Inside Windows NT, if you want "version 1" of the Internals series. Or read the Windows NT OS/2 Design Workbook - https://computernewb.com/~lily/files/Documents/NTDesignWorkb....

Yes, Win32 is just one personality, but a required one. OpenNT, Interix, SFU, SUA will ride alongside Win32. And of course there was the official OS/2 personality.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#12

Post by skissane »

kccqzy wrote: post #9
whalesalad wrote: post #3

I’ve been wanting to understand Darwin at this depth for a long time. Great read!

I've also wanted to understand Windows NT at this depth for a while. Skip the Win32 stuff, and discuss what's underneath it. As I understand Win32 is just one personality; there was also Windows Services for UNIX in the Windows XP days and Subsystem for UNIX-based Applications in Windows Vista. The underlying NT kernel is flexible enough to allow POSIX compliance. That would be an interesting read.

> As I understand Win32 is just one personality

Not really... although NT was designed to run multiple "personalities" (or "environment subsystems" to use the official term), relatively early in its development they decided to make Win32 the "primary" environment subsystem, with the result that the other two subsystems (OS/2 and POSIX) ended up relying on Win32 for essential system services.

I think this multiple personalities thing was the original vision but it never really took off in the way its original architects intended – although there used to be OS/2 and POSIX subsystems, Microsoft never put a great deal of effort into them, and now them are both dead, so Win32 is the only environment subsystem left.

Yes, there is WSL, but: WSL1 is not an environment subsystem in the classic NT sense – it has a radically different implementation from the old OS/2 and POSIX subsystems, a "picoprocess provider". And WSL2 is just a Linux virtual machine.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#13

Post by inkyoto »

bch wrote: post #6

> Mach’s virtual memory (VM) system was influential beyond the project – it was adopted by 4.4BSD and later FreeBSD as their memory management subsystem. …and NetBSD[0], OpenBSD[1], but apparently not DragonFly BSD[2]. [0] https://netbsd.org/docs/kernel/uvm.html [1] https://man.openbsd.org/OpenBSD-3.0/uvm.9 [2] https://www.dragonflybsd.org/mailarchive/kernel/2011-04/msg0...

Sadly, that is not entirely correct.

Whilst all three BSDs (386BSD, FreeBSD, and NetBSD; there was no OpenBSD in the beginning) did inherit the legacy Mach 2.5-style design, it did not live on in FreeBSD, whose core team started pretty quickly replacing all remaining vestiges of the Mach VM[0] with a complete, modern, and highly performant rewrite of the entire VM. FreeBSD 4 had none of the original Mach code left in the kernel codebase, and that happened in the late 1990s. Therefore, FreeBSD can't be referenced in a relationship to Mach apart from the initial separation/very early foundation stage.

NetBSD (and OpenBSD) went on for a while but also quickly hit the wall with the Mach design (performance, SMP/scalability, networking) and also set out on a complete rewrite with UVM (unified virtual memory) designed and led by Chuck Cranor, who wrote his dissertation on the UVM. OpenBSD later borrowed and adopted the UVM implementation, which remains in use today.

So out of all living BSD's[1], only XNU/Darwin continues to use Mach, and not Mach 2.5 but Mach 3. There have been Mach 2.5, 3 and 4 (GNU/Hurd uses Mach 4) in existence, and the compatibility between them is rather low, and remains mostly at the overall architectural level. They are better to be treated as distinct design with shared influence.

[0] Of which there were not that many to start off with.

[1] I am not sure whether DragonBSD is dead or alive today at all.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#14

Post by mattl »

skissane wrote: post #10
swatson741 wrote: post #8

Whenever I see the Darwin kernel brought into the discussion I can't help but wonder how different things could have been if Apple had just forked Linux and ran their OS services on top of that. Especially when I think about how committed they are to Darwin it really paints a poor image in my mind. The loss that open source suffers from that, and the time and money Apple has to dedicate to this with a disproportionat…

> Whenever I see the Darwin kernel brought into the discussion I can't help but wonder how different things could have been if Apple had just forked Linux XNU is only partially open sourced – the core is open sourced, but significant chunks are missing, e.g. APFS filesystem. Forking Linux might have legally compelled them to make all kernel modules open source–which while that would likely be a positive for humanity,…

At one point NeXT considered distributing GCC under the GPL with some proprietary parts linked at first boot into the binary.

Stallman after speaking with lawyers rejected this.

https://sourceforge.net/p/clisp/clisp/ci/default/tree/doc/Wh...

Look for "NeXT" on this page.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#15

Post by threeseed »

swatson741 wrote: post #8

Whenever I see the Darwin kernel brought into the discussion I can't help but wonder how different things could have been if Apple had just forked Linux and ran their OS services on top of that. Especially when I think about how committed they are to Darwin it really paints a poor image in my mind. The loss that open source suffers from that, and the time and money Apple has to dedicate to this with a disproportionat…

1) We are talking about the late 90s, well before Ubuntu, where Desktop Linux was pretty poor in terms of features and polish.

2) Apple had no money or time to invest in rewriting NeXTStep for a completely new kernel they had no experience in. Especially when so many of the dev team was involved in sorting out Apple's engineering and tech strategy as well as all the features needed to make it more Mac like.

3) Apple was still using PowerPC at the time which NeXTStep supported but Linux did not. It took IBM a couple of years to get Linux running.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#16

Post by wtallis »

swatson741 wrote: post #8

Whenever I see the Darwin kernel brought into the discussion I can't help but wonder how different things could have been if Apple had just forked Linux and ran their OS services on top of that. Especially when I think about how committed they are to Darwin it really paints a poor image in my mind. The loss that open source suffers from that, and the time and money Apple has to dedicate to this with a disproportionat…

There was never a right time for Apple to make such a switch. NeXTSTEP predates Linux, and when it was adapted into Mac OS X, Apple couldn't afford a wholesale kernel replacement project on top of everything else, and Linux in the late 1990s was far from being an obviously superior choice. Once they were a few versions in to OS X and solidly established as the most successful UNIX-like OS for consumer PCs, switching to a Linux base would have been an expensive risk with very little short-term upside.

Maybe if Apple had been able to keep classic MacOS going five years longer, or Linux had matured five years earlier, the OS X transition could have been very different. But throwing out XNU in favor of a pre-2.6 Linux kernel wouldn't have made much sense.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#18

Post by swatson741 »

wtallis wrote: post #16
swatson741 wrote: post #8

Whenever I see the Darwin kernel brought into the discussion I can't help but wonder how different things could have been if Apple had just forked Linux and ran their OS services on top of that. Especially when I think about how committed they are to Darwin it really paints a poor image in my mind. The loss that open source suffers from that, and the time and money Apple has to dedicate to this with a disproportionat…

There was never a right time for Apple to make such a switch. NeXTSTEP predates Linux, and when it was adapted into Mac OS X, Apple couldn't afford a wholesale kernel replacement project on top of everything else, and Linux in the late 1990s was far from being an obviously superior choice. Once they were a few versions in to OS X and solidly established as the most successful UNIX-like OS for consumer PCs, switching…

I agree with all of this. Moreover depending on what Torvalds chooses to do Apple may have ended up with a more expensive XNU in the end which would have been a disaster. Although I think Apple can deal with Torvalds just fine who really knows how that would have played out.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#19

Post by o11c »

inkyoto wrote: post #13
bch wrote: post #6

> Mach’s virtual memory (VM) system was influential beyond the project – it was adopted by 4.4BSD and later FreeBSD as their memory management subsystem. …and NetBSD[0], OpenBSD[1], but apparently not DragonFly BSD[2]. [0] https://netbsd.org/docs/kernel/uvm.html [1] https://man.openbsd.org/OpenBSD-3.0/uvm.9 [2] https://www.dragonflybsd.org/mailarchive/kernel/2011-04/msg0...

Sadly, that is not entirely correct. Whilst all three BSDs (386BSD, FreeBSD, and NetBSD; there was no OpenBSD in the beginning) did inherit the legacy Mach 2.5-style design, it did not live on in FreeBSD, whose core team started pretty quickly replacing all remaining vestiges of the Mach VM[0] with a complete, modern, and highly performant rewrite of the entire VM. FreeBSD 4 had none of the original Mach code left in…

> I am not sure whether DragonBSD is dead or alive today at all.

It seems to have about the same level of activity as NetBSD. Take that how you will.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#20

Post by inkyoto »

emchammer wrote: post #17

Couldn't Apple have used ZFS instead of inventing APFS? Maybe modifying it to use less physical memory?

Supporting ZFS in a UNIX kernel requires excessively extensive modifications to the design and implementation of the VMM, namely:

  1. Integration of the kernel's VM with ZFS's adaptive replacement cache which runs in user space – memory pressure cooperation, page accounting and unified memory management. It also requires extensive VM modifications to support ZFS's controlled page eviction, fine-grained dirty page tracking, plus other stuff.

  2. VMM alignment with the ZFS transactional semantics and intent logs – delayed write optimisations, efficient page syncing.

  3. Support for large memory pages and proper memory page alignment – support for superpages (to reduce the TLB pressure and to efficiently map large ZFS blocks efficiently) and I/O alignment awareness (to ensure proper alignment of memory pages to avoid unnecessary copies).

  4. Memory-mapped I/O: different implementation of mmap and support for lazy checksumming for mmap pages.

  5. Integration with kernel thread management and scheduling, co-opertation with VMM memmory allocators.

  6. … and the list goes on and on.
ZFS is just not the right answer for consumer facing and mobile/portable devices due being a heavyweight server design with vastly different design provisions and due to being the answer to a entirely different question.
Post reply on HN