21–30 of 84 posts
Related, My VM is Lighter (and Safer) than your Container (2017) : https://news.ycombinator.com/item?id=32764501 Looks like Ignite could be the best of both worlds, use the network effect of Docker images but have them as a VM instead.
Related, My VM is Lighter (and Safer) than your Container (2017) : https://news.ycombinator.com/item?id=32764501 Looks like Ignite could be the best of both worlds, use the network effect of Docker images but have them as a VM instead.
What I want to know is how I read all these posts about starting VMs in … could be I'm stuck in Azure, but I feel like I get comparable times in GCP, too.
I used to run a “bare metal servers as a service” product, and we could provision most images to a bare metal server in under 5 minutes.
If I had a VM take more than 60 seconds to come up today I’d suspect something was wrong.
Earlier quoted context omitted.
> It's hard to think of things No, it isn't. Investigate the K8S world and observe the backflips and somersaults being performed there to deal with networking and storage. Much of that is a consequence of people trying make containers do things that should be run in VMs. A full featured Linux kernel and certain key user space tools can obviate much of that nonsense with ease, deliver better performance, use less reso…
If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one? Firecracker runs a full-featured Linux kernel. It's a hypervisor. It's not a kind of kernel.
I already did that with live migration. But ok.
Encrypted storage: https://github.com/firecracker-microvm/firecracker/issues/65...: WONTFIX
The answer given is appropriate for firecracker use cases but insufficient otherwise. I'm not anti-firecracker; it's the right choice for many things. Just not all things.
The sort of VM I want orchestrated has encrypted (by contract) multi-pathed network block devices to encrypted storage volumes. 3-10 per tenant. This is trivial for a full-featured kernel; multi-path just works, encryption just works.
VLAN: https://github.com/weaveworks/ignite/issues/810: Open. Maybe one day.
Again, trivial for a full-featured Linux kernel. Has been for ages.
I think you're missing the point. It's not about what hypothetical thing firecracker can or can't do. It's about elevating VM orchestration to some degree of parity with what has been created for container orchestration. These VMs and their complex storage and networking requirements should be modeled as we model containers now; through an orchestration system that makes management easy and as foolproof as possible. The fact that firecracker isn't sufficient to be the Micro-VM of choice for this isn't relevant.
Earlier quoted context omitted.
From https://firecracker-microvm.github.io/ : Firecracker is an alternative to QEMU that is purpose-built for running serverless functions and containers safely and efficiently, and nothing more. Firecracker is written in Rust, provides a minimal required device model to the guest operating system while excluding non-essential functionality (only 5 emulated devices are available: virtio-net, virtio-block, virtio-vsoc…
Can you be specific about what those hardware things are, and why they matter for serverside workloads? Yes: the premise of Firecracker is that the hypervisor only supports the virtio devices, and doesn't emulate real hardware. Where does that end up being problematic? I'm not sure what you mean by your "CPU and memory" being "limited", either.
- Firecracker/KVM can introduce some CPU overhead, for which there are some workarounds (https://github.com/firecracker-microvm/firecracker/blob/main...)
- Firecracker networking uses a tap device. Want any kind of advanced networking hardware, say, to offload packet processing, do network inspection, etc, and access it from the containerized app? Not gonna work... IPSec/IKE VPN? May not work (iirc this requires some specific paravirt features? I might be misremembering). Have some advanced network controller that transparently handles mesh networks, or some other fancy shmancy system designed to manage complex interactions between containers and networks? Probably not gonna work due to assumptions about what lies between the layers, what components use what tricks to handle advanced routing (Netfilter, eBGP, sidecars, etc). To say nothing of link-level differences (what if your network isn't Ethernet?). And all traffic is copied from an I/O thread of an emulated network device to a host TAP device, before it makes it to the real device; my guess is network latency and maximum PPS were not a priority.
- The exposed CPU is based on what KVM supports exposing to guests; presumably QEMU supports a much wider selection. Clocksources, of which I know absolutely nothing (:D), are only allowed as kvm-clock.
- All I/O is rate-limited by a custom scheme built by the Firecracker authors. I'm sure this is fine for most use-cases, but some weird high-performance outlier is gonna hit some sort of bottleneck with this thing I'm sure.
- Firecracker emulated block devices are backed by files on the host. Ergo, any app that wants to control a disk directly, or use some fancy shmancy SAN directly, etc is out of luck.
- The guest requires a balloon driver to use balloon support, which means the guest needs this special software, and compromising the guest driver could be a serious issue. I don't know if Kata does this differently.
- Aarch64 support has a bunch of errata currently, so x86_64 is the only fully-supported platform; I dunno if Kata does any better, but this is a real hardware limitation, esp. if you're trying to buy a shit-ton of cheap powerful machines.
- This is not hardware-specific, but Firecracker seems limited to specific kernel releases; right now the latest it supports is 5.10, according to https://github.com/firecracker-microvm/firecracker/blob/main... They emphasize that they want your host and guest to run "a supported kernel", even if it's possible to run different ones. To me that says that there's the potential for Firecracker-specific bugs in newer (or older?) guest kernels. From that page: "Firecracker represents a component in a larger stack, one in which it is tightly coupled with the guest and host kernels on which it is run."
I would add that containers are used in far more settings than server-side, and would be great to have on the edge, in IoT, and on desktops, if they were a little less... funky. In general, containers requires a lot of extra steps to be "usable" as general purpose applications, and access to hardware will absolutely be a barrier we need to cross in order for "general purpose containerized apps" to become commonplace.
Earlier quoted context omitted.
If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one? Firecracker runs a full-featured Linux kernel. It's a hypervisor. It's not a kind of kernel.
> If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one? I already did that with live migration. But ok. Encrypted storage: https://github.com/firecracker-microvm/firecracker/issues/65... : WONTFIX The answer given is appropriate for firecracker use cases but insufficient otherwise. I'm not anti-firecracker; it's the right choice for many things. Just not a…
You can do all the standard Linux tap interface networking with Firecracker; it just presents as virtio ethernet to the guest.
This is the second time you've compared Firecracker to a "full-featured Linux kernel". Again: Firecracker is a hypervisor. It's not a kernel. It runs Linux kernels. "Full-featured" Linux kernels. Whatever kernel you compile.
Earlier quoted context omitted.
Can you be specific about what those hardware things are, and why they matter for serverside workloads? Yes: the premise of Firecracker is that the hypervisor only supports the virtio devices, and doesn't emulate real hardware. Where does that end up being problematic? I'm not sure what you mean by your "CPU and memory" being "limited", either.
I don't have direct experience with any of this, but just to hypothesize: - Firecracker/KVM can introduce some CPU overhead, for which there are some workarounds ( https://github.com/firecracker-microvm/firecracker/blob/main... ) - Firecracker networking uses a tap device. Want any kind of advanced networking hardware, say, to offload packet processing, do network inspection, etc, and access it from the containerized…
All serverside cloud-style VMs get tap devices. That's not a Firecracker thing.
IPSEC works just fine from within a VM. Firecracker doesn't care; it's just a hypervisor.
I'm not sure what you're trying to say about the CPU thing; Firecracker is a hypervisor, not an emulator. Linux QEMU VMs are KVM, too.
SANs work fine from within VMs. The point of a SAN is that the disk isn't attached. Firecracker talks to host block devices the same way other hypervisors do.
Here's `uname -a` from a Firecracker:
Linux 0a581153 5.12.2 #1 SMP Thu Jun 30 19:35:04 UTC 2022 x86_64 GNU/Linux
Earlier quoted context omitted.
> If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one? I already did that with live migration. But ok. Encrypted storage: https://github.com/firecracker-microvm/firecracker/issues/65... : WONTFIX The answer given is appropriate for firecracker use cases but insufficient otherwise. I'm not anti-firecracker; it's the right choice for many things. Just not a…
You can encrypt a drive from within Firecracker trivially. It's just Linux, and they're just block devices. You can do all the standard Linux tap interface networking with Firecracker; it just presents as virtio ethernet to the guest. This is the second time you've compared Firecracker to a "full-featured Linux kernel". Again: Firecracker is a hypervisor. It's not a kernel. It runs Linux kernels. "Full-featured" Linu…
This is a great idea and something I would love to use, but it's a lot less docker compatible and user friendly than it may seem. For instance there is no way to automatically run the image's specified command, effectively leaving you with a dead VM: https://github.com/weaveworks/ignite/issues/874 . You also simply can not share directories with the host (out of scope for Firecracker). microVMs for running containers…
I think my greybeard must be getting too long...
This sounds like insanity to me.
Why not just make a base image and some simple setup scripts?
If you need to get super fancy, these guys have it figured out [1]. They can clone a running vm in under two seconds.
[1] https://codesandbox.io/post/how-we-clone-a-running-vm-in-2-s...
Earlier quoted context omitted.
You can encrypt a drive from within Firecracker trivially. It's just Linux, and they're just block devices. You can do all the standard Linux tap interface networking with Firecracker; it just presents as virtio ethernet to the guest. This is the second time you've compared Firecracker to a "full-featured Linux kernel". Again: Firecracker is a hypervisor. It's not a kernel. It runs Linux kernels. "Full-featured" Linu…
Well now you're arguing with the developers of both firecracker and ignite. Maybe go to the issues I linked and correct them.
The other link you provided is to an Ignite ticket that's talking about integrating Firecracker with Linux namespaces. That has nothing to do with what Firecracker itself is capable of doing. If you want to slag Ignite, that's fine with me; I don't know much about the project, and am just here because weird things are being said about Firecracker, like that it doesn't run "full Linux kernels".