Live data from Hacker News

Apple’s Darwin OS and XNU Kernel Deep Dive

tansanrao.com

111–120 of 186 posts

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#111

Earlier quoted context omitted.

Having used continuously both FreeBSD and Linux, wherever they are best suited, since around 1995 until today, I disagree. In my opinion the single factor that has contributed the most to a greater success for Linux than for FreeBSD has been the transition to multithreaded and multicore CPUs even in the cheapest computers, which has started in 2003 with the SMT Intel Pentium 4, followed in 2005 by the dual-core AMD C…

Whilst I do agree with most of your insights and the narrative of historic events, I also believe that BSD core teams were a major contributing factor to the demise of BSD's (however unpopular such an opinion might be). The first mistake was that all BSD core teams flatly refused to provide native support for the JVM back in its heyday. They eventually partially conceded and made it work using Linux emulation; howeve…

Docker was created in 2013, long after BSDs had lost all their popularity. And, fwiw, FreeBSD pioneered containers long before Linux: https://en.m.wikipedia.org/wiki/FreeBSD_jail

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#112

Earlier quoted context omitted.

> As I understand Win32 is just one personality Not really... although NT was designed to run multiple "personalities" (or "environment subsystems" to use the official term), relatively early in its development they decided to make Win32 the "primary" environment subsystem, with the result that the other two subsystems (OS/2 and POSIX) ended up relying on Win32 for essential system services. I think this multiple per…

"At the same time, NT (up to and including Windows 2000) shipped with an OS/2 subsystem which ran character-mode 16-bit OS/2 applications." From OS/2 museum.

Turns out this is not true. Confirmed that os/2 2.0 was a skinning and compatibility layer for NT it came out for OS/2, not with windows and not from Microsoft. It came with OS/2 and from IBM. No idea whether it supported HPFS+ but it was not a subsystem.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#113
post #111

Earlier quoted context omitted.

Whilst I do agree with most of your insights and the narrative of historic events, I also believe that BSD core teams were a major contributing factor to the demise of BSD's (however unpopular such an opinion might be). The first mistake was that all BSD core teams flatly refused to provide native support for the JVM back in its heyday. They eventually partially conceded and made it work using Linux emulation; howeve…

Docker was created in 2013, long after BSDs had lost all their popularity. And, fwiw, FreeBSD pioneered containers long before Linux: https://en.m.wikipedia.org/wiki/FreeBSD_jail

FreeBSD jails are advanced chroot++. Albeit they do set a precedent for a predessor of true containers, they have:

  1. Minimal kernel isolation.

  2. Optional network stack isolation via VNET (but not used by default).

  3. Rudimentary resource controls with no default enforcement (important!).

  4. Simple capability security model.
Most importantly, since FreeBSD was a very popular choice for hosting providers at the time, jails were originally invented to fully support partitioned-off web hosting, rather than to run self-sufficient, fully contained (containerised) applications as first-class citizens.

The claim to have invented true containers belongs to Solaris 10 (not Linux) and its zones. Solaris 10 was released in January 2005.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#114
What impresses me most about technical documentation like this is how it structures knowledge into comprehensible layers. This article manages to explain an extremely complex system by establishing clear relationships between components.

I've been experimenting with similar approaches for documentation in open source projects, using knowledge graphs to link concepts and architectural decisions. The biggest challenge is always keeping documentation synchronized with evolving code.

Has anyone found effective tools for maintaining this synchronization between documented architecture and implemented code? Large projects like Darwin must have established processes for this.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#115
post #111

Earlier quoted context omitted.

Docker was created in 2013, long after BSDs had lost all their popularity. And, fwiw, FreeBSD pioneered containers long before Linux: https://en.m.wikipedia.org/wiki/FreeBSD_jail

FreeBSD jails are advanced chroot++. Albeit they do set a precedent for a predessor of true containers, they have: 1. Minimal kernel isolation. 2. Optional network stack isolation via VNET (but not used by default). 3. Rudimentary resource controls with no default enforcement (important!). 4. Simple capability security model. Most importantly, since FreeBSD was a very popular choice for hosting providers at the time,…

[deleted]

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#116
post #103
post #53

Earlier quoted context omitted.

You haven’t really expanded on why basing off the Linux kernel would have made more sense, especially at the time. People have responded to you with timelines explaining why it couldn’t have happened but you seem to keep restating this claim without more substance or context to the time. Imho Linux would have been the wrong choice and perhaps even the incorrect assumption. Mac is not really BSD based outside of the u…

Well for starters, it would have better memory management. The XNU kernel's memory manager has poor time complexity. If I create a bunch of sparse memory maps using mmap() then XNU starts to croak once I have 10,000+ of them.

Please re read the comment you’re responding to about how the kernel would have diverged significantly even if they did use the Linux kernel. Unless you think a three decade old kernel would have the same characteristics as today.

What benefit would it have had at the time? What guarantees would it have given at the time that would have persisted three decades later?

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#117
post #60
post #35

Earlier quoted context omitted.

Based on how often they pull in updated bits from FreeBSD (pretty much never), an Apple fork of Linux would be more or less Linux 2.4 today. I don't know what the loss that open source suffers is in this context? I don't think Apple would need to spend less time or money on their kernel grafted ontop of Linux 2.4 vs their kernel grafted on top of FreeBSD 4.4

Because presumably the GPL would force them to release their modifications. Apple gets/got away with leeching off the BSDs because of the permissive license.

They release their kernel source more or less timely without the GPL.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#118

Whenever I see the Darwin kernel brought into the discussion I can't help but wonder how different things could have been if Apple had just forked Linux and ran their OS services on top of that. Especially when I think about how committed they are to Darwin it really paints a poor image in my mind. The loss that open source suffers from that, and the time and money Apple has to dedicate to this with a disproportionat…

Control is important. Apple has never had to fight with Torvalds or IBM or Microsoft over getting something added to the kernel. Just look at the fiasco when Microsoft wanted to add a driver for their virtualization system to the kernel.

Also, one thing you'll notice about big companies - they know that not only is time valuable, worst-case time is important too. If someone in an open-source ecosystem CAN delay your project, that's almost as bad as if they regularly DO delay your project. This is why big companies like Google tend to invent everything themselves, I.E. Google may have "invented Kubernetes" (really, an engineer at Google uninvolved with the progenitor of K8s - Borg - invented it based on Borg), but they still use Borg, which everyone Xoogler here likes to say is "not as good as k8s". Yet they still use it. Because it gives them full control, and no possibility of outsiders slowing them down.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#119
post #80

Earlier quoted context omitted.

> As things now stand, FreeBSD represents many of the benefits of Darwin and the open source nature of Linux. No. FreeBSD has committed the original sin of UNIX by deliberately dropping support for all non-Intel architectures, intending to focus on optimising FreeBSD for the Intel ISA and platforms. UNIX portability and support for a diverse range of CPU's and hardware platforms are ingrained in the DNA of UNIX, howe…

Having used continuously both FreeBSD and Linux, wherever they are best suited, since around 1995 until today, I disagree. In my opinion the single factor that has contributed the most to a greater success for Linux than for FreeBSD has been the transition to multithreaded and multicore CPUs even in the cheapest computers, which has started in 2003 with the SMT Intel Pentium 4, followed in 2005 by the dual-core AMD C…

I don't know if this had much affect on anything, but another thing that hindered using FreeBSD for some users was that Linux worked better as a dual boot system with DOS/Windows on a typical home PC.

There were two problems.

The first was that FreeBSD really wanted to own the whole disk. If you wanted to dual boot with DOS/Windows you were supposed to put FreeBSD on a separate disk. Linux was OK with just having a partition on the same disk you had DOS/Windows on. For those of us whose PCs only had one hard disk, buying a copy of Partition Magic was cheaper than buying a second hard disk.

The reason for this was that the FreeBSD developers felt that multiple operating system on the same disk was not safe due to the lack of standards for how to emulate a cylinder/head/sector (CHS) addressing scheme on disks that used logical block addressing (LBA). They were technically correct, but greatly overestimated the practical risks.

In the early days PC hard disks used CHS addressing, and the system software such as the PC BIOS worked in those terms. Software using the BIOS such as DOS applications and DOS itself worked with CHS addresses and the number of cylinders, heads, and sectors per track (called the "drive geometry") they saw matched the actual physical geometry of the drive.

The INT 13h BIOS interface for low level disk access allowed for a maximum of 1024 cylinders, 256 heads, and 63 sectors per track (giving a maximum possible drive size of 8 GB if the sectors were 512 bytes).

At some point as disks got bigger drives with more than 63 sectors per track became available. If you had a drive with for example 400 cylinders, 16 heads, and 256 sectors per track you would only be able to access about 1/4 of the drive using CHS addressing that uses the actual drive geometry.

It wasn't really practical to change the INT 13h interface to give the sectors per track more bits, and so we entered the era of made up drive geometries. The BIOS would see that the disk geometry is 400/16/256 and make up a geometry with the same capacity that fit within the limits, such as 400/256/16.

Another place with made up geometry was SCSI disks. SCSI used LBA addressing. If you had a SCSI disk on your PC whatever implemented INT 13h handling for that (typically the BIOS ROM on your SCSI host adaptor) would make up a geometry. Different host adaptor makers might use different algorithms for making up that geometry. Non-SCSI disk interfaces for PCs also moved to LBA addressing, and so the need to make up a geometry for INT 13h arose with those too, and different disk controller vendors might use a different made up geometry.

So suppose you had a DOS/Windows PC, you repartitioned your one disk to make room for FreeBSD, and went to install FreeBSD. FreeBSD does not use the INT 13h BIOS interface. It uses its own drivers to talk to the low level disk controller hardware and those drivers use LBA addressing.

It can read the partition map and find the entry for the partition you want to install on. But the entries in the partition map use CHS addressing. FreeBSD would need to translate the CHS addresses from the partition map into LBA addresses, and to do that it would need to know the disk geometry that whatever created the partition map was using. If it didn't get that right and assumed a made up geometry that didn't match the partitioner's made up geometry the actual space for DOS/Windows and the actual space for FreeBSD could end up overlapping.

In practice you can almost always figure out from looking at the partition map what geometry the partitioner used with enough accuracy to avoid stomping on someone else's partition. Partitions started at track boundaries, and typically the next partition started as close as possible to the end of the previous partition and that sufficiently narrows down where the partition is supposed to be in LBA address space.

That was the approach taken by most SCSI vendors and it worked fine. I think eventually FreeBSD did start doing this too but by then Linux had become dominant in the "Dual boot DOS/Windows and a Unix-like OS on my one disk PC" market.

The other problem was CD-ROM support. FreeBSD was slow to support IDE CD-ROM drives. Even people who had SCSI on their home PC and used SCSI hard disks were much more likely to have an IDE CD-ROM than a SCSI CD-ROM. SCSI CD-ROM drives were several times more expensive and it wasn't the interface that was the bottleneck so SCSI CD-ROM just didn't make much sense on a home PC.

For many then it came down to with Linux they could install they didn't need a two disk system and they could install from a convenient CD-ROM, but for FreeBSD they would need a dedicated disk for it and would have to deal with a stack of floppies.

Re: Apple’s Darwin OS and XNU Kernel Deep Dive

#120

Whenever I see the Darwin kernel brought into the discussion I can't help but wonder how different things could have been if Apple had just forked Linux and ran their OS services on top of that. Especially when I think about how committed they are to Darwin it really paints a poor image in my mind. The loss that open source suffers from that, and the time and money Apple has to dedicate to this with a disproportionat…

>Whenever I see the Darwin kernel brought into the discussion I can't help but wonder how different things could have been if Apple had just forked Linux and ran their OS services on top of that.

They have a long history with XNU and BSD. And Linux has s GPL license which might not suit Apple.

>Especially when I think about how committed they are to Darwin it really paints a poor image in my mind. The loss that open source suffers from that, and the time and money Apple has to dedicate to this with a disproportionate return.

They share a lot of code with FreeBSD, NetBSD and OpenBSD. Which are open source. And Darwin is open source, too. So there's no loss that open source suffers.

Post reply on HN