Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My First Unikernel

roscidus.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: My First Unikernel

#31

Post by avsm »

frik wrote: post #20
enduser wrote: post #5

If it helps to add clarity to what this is about, Xen still requires a host (dom0) operating system, usually Linux but also NetBSD and OpenSolaris. Mirage OS (the basis for building your own unikernel) allows building applications as domU kernels. This is advantageous because it minimizes context-switching and allows all of the code in a Xen VM (domU) to be written in a safer language than C. So the dom0 OS (Linux) s…

Even though Amazon EC2 is Xen-based (internally), you have to run a lightweight Linux distro to use Mirage on AWS EC2 (at least for now), as I learned from the MirageOS posting a few weeks ago.

(mirage dev here)

Not true -- you can run Mirage Xen kernels directly as EC2 guest VMs by using their AKI feature to boot a custom kernel. Instead of a custom Linux kernel, just boot a unikernel and don't bother with user space :-)

Go to http://github.com/mirage/mirage and check out scripts/ec2.sh for a shell script that automates this (needs to be more user friendly, patches welcome)

Re: My First Unikernel

#32

Post by avsm »

mwcampbell wrote: post #19
amirmc wrote: post #16

Earlier quoted context omitted.

Even if you have your own hardware there are scenarios where you might want stronger isolation, increased density and/or heterogeneous deployments. All of which is achievable with Unikernels running on Xen.

How would unikernels running on Xen give you better density than containers (which are really just namespaced processes) running on a shared Unix kernel on bare metal? Wouldn't the Xen-based approach have more overhead?

This isn't supported yet, but unikernels on Xen don't currently switch address spaces (cr2 on x86). We're thinking about how to increase density by supporting process switching on Xen, which isn't hard, but needs to be done carefully.

Xen also supports memory sharing among VMs if running in a hardware container, and Hwanju Kim added support for "PVH" mode to MiniOS recently which unlocks this functionality in Mirage. Early days on this work, but fun ones, since we aren't bound by the compatibility constraints of Linux containers, but have access to the same hardware resources.

Re: My First Unikernel

#33

Post by baruch »

paulasmuth wrote: post #6

This sounds like an awesome and fun project to hack on! However, optimizing around context switches and task preemptions is something you would usually do if your application is actually bound by IO/context switching, is extremely latency sensitive or when you are trying to squeeze the last bits of performance out of a machine. Why did you choose to build such a microoptimized system in a garbage collected language?…

OSv does something similar to Mirage it seems and is written in C++. They also can run different languages, I believe I saw lua support and they commercially target Java too.

Re: My First Unikernel

#34

Post by avsm »

616c wrote: post #7

So I have been following this stuff with a lot of interest, but I am not very familiar with Ocaml and have tried looking at Mirage docs with some limited downtime. If Ocaml can only handle one core and does not do SMP, how does it do in the cloud? Does this mean Mirage unikernels handle only one processor/core in Amazon and elsewhere?

Two answers: scaling through structured distributed systems abstractions is a key aim in Mirage. We deliberately want each VM to be predictable, single vCPU and scale via multiple VMs. It's far more efficient to scale via lots of small VMs that are scheduled independently than having a few big multicpu VMs (which have a lot of overhead since the CPU sync also needs to be virtualised). See our ASPLOS 2013 paper for some simple workloads on this topic.

We are building this library to simplify this style of distributed programming: http://openmirage.org/blog/introducing-irmin

Other answer: we will be talking about our multicore ocaml implementation at ICFP in September. I still don't want to see it in Mirage though :-)

Re: My First Unikernel

#35

Post by avsm »

paulasmuth wrote: post #18
andolanra wrote: post #15

Earlier quoted context omitted.

Don't think of it as optimizing around context switches—think of it as just omitting what you don't need, and losing context-switches as a side benefit. A multi-user OS has a lot of stuff which might not be necessary for a single virtualized service (various security mechanisms, lots of file system niceties, running other services, &c), so writing a unikernel like this allows you to select exactly as much as you need…

> so writing a unikernel like this allows you to select exactly as much as you need for your particular service. I agree this is a big upside of the authors approach. Less dependencies lead to fewer problems caused by external/upstream changes. > OCaml isn't all that much slower than C++. To use the Programming Language Shootout as a rough esimation[^1], it can even come close to matching C++ in certain programs, and…

Regarding GC pauses, remember that you already have them if your application stack is written in (Scala,Java,Go,OCaml,Haskell). But you also have other manual memory management going on everywhere!

With Mirage, it's all amortised in one consistent, fast GC! To give you a sense of the malloc vs OCaml GC trade off, see http://anil.recoil.org/papers/2007-eurosys-melange.pdf

Malloc and free list management is remarkably complex compared to a fast, simple GC. It would be interesting to build an OCaml runtime in Rust to experiment with these tradeoffs in a more controlled fashion.

Re: My First Unikernel

#36

Post by wmf »

avsm wrote: post #34
616c wrote: post #7

So I have been following this stuff with a lot of interest, but I am not very familiar with Ocaml and have tried looking at Mirage docs with some limited downtime. If Ocaml can only handle one core and does not do SMP, how does it do in the cloud? Does this mean Mirage unikernels handle only one processor/core in Amazon and elsewhere?

Two answers: scaling through structured distributed systems abstractions is a key aim in Mirage. We deliberately want each VM to be predictable, single vCPU and scale via multiple VMs. It's far more efficient to scale via lots of small VMs that are scheduled independently than having a few big multicpu VMs (which have a lot of overhead since the CPU sync also needs to be virtualised). See our ASPLOS 2013 paper for so…

It's far more efficient to scale via lots of small VMs that are scheduled independently than having a few big multicpu VMs (which have a lot of overhead since the CPU sync also needs to be virtualised).

Are you assuming that vCPUs are scheduled or pinned? It wasn't clear from skimming the paper. I would agree that two levels of scheduling are bad but if that's the problem pinning should fix it.

The situation may change if you eliminate the hypervisor. My intuition is that a single multi-threaded process with work-stealing will be faster than separate processes or VMs due to better load balance (see PX vs. 1X dynos) and faster inter-thread communication (if your app has any communication).

Re: My First Unikernel

#37

Post by derefr »

dkarapetyan wrote: post #3

I'm a little confused about the direction things are going in. I like high level languages and I like that the OS manages certain resources so I don't have to. This guy is writing directly to block devices from OCaml. Don't get me wrong it's all pretty cool but there is some kind of dissonance there I can't reconcile. Is Xen the new OS now?

Note, though, that he is indeed writing to a block device--which is much higher-level than writing to a disk.

Remember back in the 80s, when the BIOS was actually an effective hardware abstraction layer--giving you a defined interrupt to ask the BIOS to, say, write to a disk--and the OS was just for module loading and scheduling and policy-based security? (Not that DOS did either of the latter.)

Well, Xen isn't the new OS; instead, the domU is the new BIOS, and hypercalls are the new BIOS interrupts.

I really hope to see Linux redone (or another *nix created) in this "unikernel" style, where everything hardware-like or HAL-like is taken out, and instead things like filesystem drivers are implemented directly in terms of hypercalls.

I also hope to see UEFI reimplemented as a resident domU, such that a plain old desktop or notebook computer could treat its user OS as a container-image to be slung around, rather than having it "own" the hardware. UEFI actually already supports this mode of operation--allowing you to boot "UEFI applications" that keep UEFI around to provide BIOS-like functionality--but I don't know of a single OS that makes use of that, rather than overwriting the processor interrupt vectors and claiming all of physical memory for itself.

Re: My First Unikernel

#38

Post by avsm »

wmf wrote: post #36
avsm wrote: post #34

Earlier quoted context omitted.

Two answers: scaling through structured distributed systems abstractions is a key aim in Mirage. We deliberately want each VM to be predictable, single vCPU and scale via multiple VMs. It's far more efficient to scale via lots of small VMs that are scheduled independently than having a few big multicpu VMs (which have a lot of overhead since the CPU sync also needs to be virtualised). See our ASPLOS 2013 paper for so…

It's far more efficient to scale via lots of small VMs that are scheduled independently than having a few big multicpu VMs (which have a lot of overhead since the CPU sync also needs to be virtualised). Are you assuming that vCPUs are scheduled or pinned? It wasn't clear from skimming the paper. I would agree that two levels of scheduling are bad but if that's the problem pinning should fix it. The situation may chan…

If you eliminate the hypervisor, you don't have vCPUs at all, so I'm not sure how this is a useful comparison.

The question of IPC performance is an interesting one. We've been building up a database of open source results that show wildly diverging results on different architectures. Surf through http://fable.io for it, or read this work in progress:

http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1.... http://anil.recoil.org/talks/fosdem-io-2012.pdf (fosdem slides)

The TL;DR of these numbers is that it's very hard to make firm performance hypotheses about IPC across architectures, NUMA and hypervisors.

Re: My First Unikernel

#39

Post by avsm »

derefr wrote: post #37
dkarapetyan wrote: post #3

I'm a little confused about the direction things are going in. I like high level languages and I like that the OS manages certain resources so I don't have to. This guy is writing directly to block devices from OCaml. Don't get me wrong it's all pretty cool but there is some kind of dissonance there I can't reconcile. Is Xen the new OS now?

Note, though, that he is indeed writing to a block device--which is much higher-level than writing to a disk . Remember back in the 80s, when the BIOS was actually an effective hardware abstraction layer--giving you a defined interrupt to ask the BIOS to, say, write to a disk--and the OS was just for module loading and scheduling and policy-based security? (Not that DOS did either of the latter.) Well, Xen isn't the…

We have a feature issue tracking UEFI in Mirage, but noone's written the bootloader code yet: https://github.com/mirage/mirage/issues/187

Interestingly, it looks like the easiest way to support Azure...

Re: My First Unikernel

#40

Post by xxpor »

derefr wrote: post #37
dkarapetyan wrote: post #3

I'm a little confused about the direction things are going in. I like high level languages and I like that the OS manages certain resources so I don't have to. This guy is writing directly to block devices from OCaml. Don't get me wrong it's all pretty cool but there is some kind of dissonance there I can't reconcile. Is Xen the new OS now?

Note, though, that he is indeed writing to a block device--which is much higher-level than writing to a disk . Remember back in the 80s, when the BIOS was actually an effective hardware abstraction layer--giving you a defined interrupt to ask the BIOS to, say, write to a disk--and the OS was just for module loading and scheduling and policy-based security? (Not that DOS did either of the latter.) Well, Xen isn't the…

>I also hope to see UEFI reimplemented as a resident domU, such that a plain old desktop or notebook computer could treat its user OS as a container-image to be slung around, rather than having it "own" the hardware. UEFI actually already supports this mode of operation--allowing you to boot "UEFI applications" that keep UEFI around to provide BIOS-like functionality--but I don't know of a single OS that makes use of that, rather than overwriting the processor interrupt vectors and claiming all of physical memory for itself.

This is really the opposite direction things are going, especially in x86_64. PV is horrendously inefficient in 64 bit mode, because of the removal of CPU ring 1 and 2.

http://wiki.xen.org/wiki/Virtualization_Spectrum#Problems_wi...

HVM allows PCI passthrough if your CPU and chipset support it, which means domU now has direct access to the hardware, with no dom0/qemu layer to get in the way and slow things down.

Post reply on HN