Ignite – Use Firecracker VMs with Docker images • Hacker News Board

Re: Ignite – Use Firecracker VMs with Docker images

#31

Post by bogomipz » Tue, Sep 27, 2022, 5:30 AM UTC

tptacek wrote: post #27

0xbadcafebee wrote: post #24

Earlier quoted context omitted.

I don't have direct experience with any of this, but just to hypothesize: - Firecracker/KVM can introduce some CPU overhead, for which there are some workarounds ( https://github.com/firecracker-microvm/firecracker/blob/main... ) - Firecracker networking uses a tap device. Want any kind of advanced networking hardware, say, to offload packet processing, do network inspection, etc, and access it from the containerized…

I'm not trying to be argumentative, but rather just to clear things up for you: All serverside cloud-style VMs get tap devices. That's not a Firecracker thing. IPSEC works just fine from within a VM. Firecracker doesn't care; it's just a hypervisor. I'm not sure what you're trying to say about the CPU thing; Firecracker is a hypervisor, not an emulator. Linux QEMU VMs are KVM, too. SANs work fine from within VMs. The…

>"All serverside cloud-style VMs get tap devices."

I'm having trouble understanding this sentence. Specifically "serverside cloud-style VMs." Does "serverside" mean from the host OS that's running the Hypervisor? Like if I were logged into that host an "ifconfig" would show me tap devices?

Re: Ignite – Use Firecracker VMs with Docker images

#32

Post by tptacek » Tue, Sep 27, 2022, 5:34 AM UTC

bogomipz wrote: post #31

tptacek wrote: post #27

Earlier quoted context omitted.

I'm not trying to be argumentative, but rather just to clear things up for you: All serverside cloud-style VMs get tap devices. That's not a Firecracker thing. IPSEC works just fine from within a VM. Firecracker doesn't care; it's just a hypervisor. I'm not sure what you're trying to say about the CPU thing; Firecracker is a hypervisor, not an emulator. Linux QEMU VMs are KVM, too. SANs work fine from within VMs. The…

>"All serverside cloud-style VMs get tap devices." I'm having trouble understanding this sentence. Specifically "serverside cloud-style VMs." Does "serverside" mean from the host OS that's running the Hypervisor? Like if I were logged into that host an "ifconfig" would show me tap devices?

Yes, it does.

Re: Ignite – Use Firecracker VMs with Docker images

#33

Post by 0xbadcafebee » Tue, Sep 27, 2022, 5:41 AM UTC

tptacek wrote: post #27

0xbadcafebee wrote: post #24

Earlier quoted context omitted.

I don't have direct experience with any of this, but just to hypothesize: - Firecracker/KVM can introduce some CPU overhead, for which there are some workarounds ( https://github.com/firecracker-microvm/firecracker/blob/main... ) - Firecracker networking uses a tap device. Want any kind of advanced networking hardware, say, to offload packet processing, do network inspection, etc, and access it from the containerized…

I'm not trying to be argumentative, but rather just to clear things up for you: All serverside cloud-style VMs get tap devices. That's not a Firecracker thing. IPSEC works just fine from within a VM. Firecracker doesn't care; it's just a hypervisor. I'm not sure what you're trying to say about the CPU thing; Firecracker is a hypervisor, not an emulator. Linux QEMU VMs are KVM, too. SANs work fine from within VMs. The…

> All serverside cloud-style VMs get tap devices

Unless you want to PCI pass-through an SRIOV VF into a guest? If you're paying out the ass for Cisco cloud gear you probably want the guest to have direct access to a card...

Again, I don't remember the specifics, but depending on the underlying networking (on the host etc), there may be some issues with IPSec (but I could be wrong). From some random searching: https://linux-ipsec.org/wp-content/uploads/slides/2018/quest...

> SANs work fine from within VMs. The point of a SAN is that the disk isn't attached. Firecracker talks to host block devices the same way other virtualizers do.

But you may want to talk to an attached HBA? Firecracker's docs (https://github.com/firecracker-microvm/firecracker/blob/main...) say: "Firecracker emulated block devices are backed by files on the host. To be able to mount block devices in the guest, the backing files need to be pre-formatted with a filesystem that the guest kernel supports."

Based on everything else in the Firecracker docs, I don't see any vHBA support provided, so the guest cannot access an HBA on the host. And a virtual device backed by a file is going to have crap performance.

Re: Ignite – Use Firecracker VMs with Docker images

#34

Post by sofixa » Tue, Sep 27, 2022, 6:37 AM UTC

claytongulick wrote: post #29

solarkraft wrote: post #8

This is a great idea and something I would love to use, but it's a lot less docker compatible and user friendly than it may seem. For instance there is no way to automatically run the image's specified command, effectively leaving you with a dead VM: https://github.com/weaveworks/ignite/issues/874 . You also simply can not share directories with the host (out of scope for Firecracker). microVMs for running containers…

> microVMs for running containers are definitely a great idea I think my greybeard must be getting too long... This sounds like insanity to me. Why not just make a base image and some simple setup scripts? If you need to get super fancy, these guys have it figured out [1]. They can clone a running vm in under two seconds. [1] https://codesandbox.io/post/how-we-clone-a-running-vm-in-2-s...

> Why not just make a base image and some simple setup scripts?

That makes it possible for you to create an image (VM or container?, good start. But what Firecracker and containers solve is isolation and ease of distribution.

Re: Ignite – Use Firecracker VMs with Docker images

#35

Post by sofixa » Tue, Sep 27, 2022, 6:41 AM UTC

deathanatos wrote: post #21

cercatrova wrote: post #2

Related, My VM is Lighter (and Safer) than your Container (2017) : https://news.ycombinator.com/item?id=32764501 Looks like Ignite could be the best of both worlds, use the network effect of Docker images but have them as a VM instead.

What I want to know is how I read all these posts about starting VMs in … could be I'm stuck in Azure, but I feel like I get comparable times in GCP, too.

Azure is famous for two things - being extremely slow (including hitting timeouts on connected software), and since a year or so ago for having trivial yet highly critical security vulnerabilities that escape the tenant barrier.

Regarding the speed, urban legend has it Azure is basically a giant mess of Powershell duct tape.

Also it's not great redundancy-wise, most regions are not composed of multiple Availability Zones, and their control plane had a reliance on a single DC(which, like their security failures, shows that redundancy and security simply aren't priorities there).

Re: Ignite – Use Firecracker VMs with Docker images

#36

Post by imiric » Tue, Sep 27, 2022, 6:53 AM UTC

claytongulick wrote: post #29

solarkraft wrote: post #8

This is a great idea and something I would love to use, but it's a lot less docker compatible and user friendly than it may seem. For instance there is no way to automatically run the image's specified command, effectively leaving you with a dead VM: https://github.com/weaveworks/ignite/issues/874 . You also simply can not share directories with the host (out of scope for Firecracker). microVMs for running containers…

> microVMs for running containers are definitely a great idea I think my greybeard must be getting too long... This sounds like insanity to me. Why not just make a base image and some simple setup scripts? If you need to get super fancy, these guys have it figured out [1]. They can clone a running vm in under two seconds. [1] https://codesandbox.io/post/how-we-clone-a-running-vm-in-2-s...

> Why not just make a base image and some simple setup scripts?

Two major reasons:

- Reusing the vast availability of prebuilt OCI images. I don't need to create scripts at all if I find a good image someone already created a Dockerfile for. Dockerfiles are scripts anyway, especially tailored for provisioning.

- Docker's main benefit is a friendly UI over low-level Linux primitives. In this case, Ignite offers a similarly familiar UI over Firecracker/KVM, which is also cumbersome to use directly.

Re: Ignite – Use Firecracker VMs with Docker images

#37

Post by topspin » Tue, Sep 27, 2022, 7:26 AM UTC

tptacek wrote: post #30

topspin wrote: post #28

Earlier quoted context omitted.

Well now you're arguing with the developers of both firecracker and ignite. Maybe go to the issues I linked and correct them.

No, you're linking to tickets that I don't think you really grok. That's someone asking for an extra encryption feature on the host side for Firecracker images, not someone saying it's impossible to encrypt a drive from within a Firecracker VM, which is obviously possible. You can just, you know, boot one up and try. The other link you provided is to an Ignite ticket that's talking about integrating Firecracker with…

No post body was provided.

Re: Ignite – Use Firecracker VMs with Docker images

#38

Post by regularfry » Tue, Sep 27, 2022, 7:37 AM UTC

mbreese wrote: post #6

topspin wrote: post #3

Earlier quoted context omitted.

I think lightweight VMs will have a major impact. VMs are powerful and solve legions of problems that arise trying to make containers match them, which is exactly the road currently being paved by so much of Kubernetes. Firecracker has a number of limitations, however. Firecracker is tailored to the highly vertical 'lambda' use case and too much of the power of the kernel and userspace is stripped away. You can't eve…

Why would you want to migrate a firecracker vm? They are supposed to be for small, short lived processes. Or in this case, Docker containers (which are likely to be controlled by some kind of orchestration). If you want to be able to migrate a running VM, do you still really need it to start super fast? It’s going to be a long lived service (otherwise you wouldn’t migrate it), so a longer spin up time should be accep…

> If you want to be able to migrate a running VM, do you still really need it to start super fast?

Yes. "You" is two different people in this sentence. The person doing the migration is often not the person who owns whatever service is running in the VM. They have uncorrelated needs. As a platform operator, I want to migrate VMs to be able to balance load and evacuate dying hardware. As a service operator on that platform I want high uptime - or, more accurately to this case, low downtime.

One of the reasons someone goes for a long-lived service on a VM platform is because the service is in some sense critical: it's Bad News when it drops a connection or doesn't respond. So the operator of that service wants something with a fast restart so that if it ever does need to be bounced, it's back quickly and they return to whatever redundancy level they originally designed to as fast as possible.

"Long uptime means a slow boot is fine" has never been true, in my experience.

Re: Ignite – Use Firecracker VMs with Docker images

#39

Post by sneak » Tue, Sep 27, 2022, 7:44 AM UTC

claytongulick wrote: post #29

solarkraft wrote: post #8

This is a great idea and something I would love to use, but it's a lot less docker compatible and user friendly than it may seem. For instance there is no way to automatically run the image's specified command, effectively leaving you with a dead VM: https://github.com/weaveworks/ignite/issues/874 . You also simply can not share directories with the host (out of scope for Firecracker). microVMs for running containers…

> microVMs for running containers are definitely a great idea I think my greybeard must be getting too long... This sounds like insanity to me. Why not just make a base image and some simple setup scripts? If you need to get super fancy, these guys have it figured out [1]. They can clone a running vm in under two seconds. [1] https://codesandbox.io/post/how-we-clone-a-running-vm-in-2-s...

General advice: when smart people are investing time and energy into a solution, and your alternative solution involves the word "just", it's usually a code smell that you're not giving the people doing the work enough credit.

Re: Ignite – Use Firecracker VMs with Docker images

#40

Post by zozbot234 » Tue, Sep 27, 2022, 7:57 AM UTC

Running Docker in a VM is not new at all. This is even how Docker for Windows/Mac works.

The GitHub page states that Ignite does more than just "wrapping a container in a VM layer" but I'm not sure how that matters, since the notion of a "container" is purely about configuration. It seems like a distinction without a difference.