How to Write an Operating System

I tried to build one from scratch based on my own L4 microkernel. Given the bloated nature of monolithic kernels and their irrelevance to embedded space, I had the thought that if I wrote something small and neat it could gain traction for embedded systems. For that I took the BSD virtual memory design, and implemented a complete VM system from scratch. Then I wrote a VFS layer but that one was partially complete. I had demand paging implemented and supporting calls like fork() execve() etc. It was challenging and fun - but an incomplete project, also my biggest error from a startup/making money point of view. Basically having a new kernel is a hard sell nowadays, no matter how neat it is. It was more of an intellectual challenge for me rather than a startup - but still I was in the expectation that it could be a success.

About exactly a year ago, I changed my direction to virtualization. I ditched about %60 of the software that includes all that VM layer and VFS for which I put loads of effort (many man months...). I realized I was out of focus, so I changed my focus to enhancing only the microkernel itself. I decided upon three specific goals: 1) Adding multi-core support for latest ARM cores (e.g. the ones that will be on most high-end mobile phones and devices) 2) Adding fine-grain security and control based on capabilities. (in short, having control over system calls) 3) Adding virtualization support for the linux kernel. Right now I am done with the first two, and about to get done with the 3rd one.

My interest has always been to create something new in the OS/kernel space. After doing much research I decided that L4 microkernel design represents the most promising work in this field. This is because a) monolithic kernels are mature and good for what they are doing b) microkernels can solve certain new problems like virtualization.

Eventually I was convinced that embedded virtualization would be the most interesting problem I could solve. In my opinion there is still room for systems-level work particularly in Virtualization. The APIs are not set in stone. Not many people know how virtualization works, let alone design an interface for it from scratch.

I think new areas like this will come up but its quite important to spot an unsolved problem because if you dive in to write a kernel its an expensive journey.

And time. Sometimes that's what the FU money would bring to the table.

My friend was part of a group of students who decided to write XOmB. They just apparently decided it would be a neat project, now it's a pretty cool exokernel: http://xomb.net/index.php?title=Main_Page

Hey, thanks for the shout-out.

Not only are we working on XOmB, (pronounced 'zombie') but we extracted all of the stuff "from power on 'till kmain" and called it XOmB Bare Bones, so that other people who want to write kernels in D don't have to worry about that ugliness: http://wiki.xomb.org/index.php?title=XOmB_Bare_Bones

That is great to hear. I've heard a few stories on how painful that stuff was to track down and write. Bring up always sucks. This sounds like an excellent way for people to dig into writing kernels without forcing people to write all the bootup code before they can do anything.

How do you do device handoff? Are people restricted to an exokernel model or can they handoff to a macrokernel kexec style?

(VMware used to use the Linux kernel to take care of init before they wrote it themselves, so there's definitely a need for this type of thing, even beyond hobbyist.)