Live data from Hacker News

Immutable Software Deploys Using ZFS Jails on FreeBSD

conradresearch.com

41–50 of 54 posts

Re: Immutable Software Deploys Using ZFS Jails on FreeBSD

#41
post #4
post #2

Isn't this just docker with extra steps?

No. ZFS has been stable in FreeBSD for something like 17 years, and FreeBSD jails have been around for something like 25 years. By the time Docker hit 1.0 (about 11 years ago), the use of snapshots and jails had already been normal parts of life in the FreeBSD space for over half of a decade.

[dead]

Re: Immutable Software Deploys Using ZFS Jails on FreeBSD

#42
I highly recommend this approach. I run ~all of my digital footprint on FreeBSD jails. After surveying how many jail managers have come and go in the last decade, I decided to just roll my own using a single shell script called jailctl [0].

Nothing fancy, just VNET jails based on ZFS templates (vanilla FreeBSD rootfs) and epair interfaces (which I truck to various VLANs on the host's egress interface).

One pattern that I've found useful is to give each jail a persistently delegated ZFS dataset called "data." This lets me reprovision the OS image for the jail without having to backup and restore its application data (such as a Postgres DB). It also allows each jail to manage its own ZFS snapshots.

The only thing that was a bit hairy was generating unique interface names and MAC addresses for each jail's VNET interface. My first instinct was to derive the interface name from the jail name, but interface names on FreeBSD are limited to 15 characters, and occasionally I'd hit this limit.

In the end I did some dark magic using md5 sums of the jail name / host interface MAC address. Kind of ugly but I really didn't want to introduce any dependencies besides /bin/sh.

[0] https://github.com/cullumsmith/infrastructure/blob/master/fi...

Re: Immutable Software Deploys Using ZFS Jails on FreeBSD

#43
I used to do something like this but with ZFS on an OpenSolaris / Illumos storage server, exporting copy-on-write clones of snapshots of iSCSI volumes to boot Xen VMs on neighbouring blades, with tagged VLANs from each host because there were multiple guests to launch with varying roles in the application cluster. We made the VLAN number match 12 bits of the IP addresses and numbered the clones similarly. It merely remained to create new snapshots every release and any dev could launch an bugfix/test/showcase/etc environment for that version, and connect to it via a VPN. I was always worried about scale if we hired more than 4096 developers but fortunately the company was acquired and its product discontinued before that happened.

That was in 2007 so the control plane (scheduler and automation) were built from scratch and we had very few reference points for the overall design. If I was building that today I’d probably still use ZFS clones but at filesystem level instead of block devices, and serve jails over NFS if I can get away with it, the iSCSI part was always a little janky.

Re: Immutable Software Deploys Using ZFS Jails on FreeBSD

#44

Earlier quoted context omitted.

I never find arguments like this compelling (but I agree with your sentiment). I don’t much care to fire up the time machine to go back 12 or more years to develop software today. If your argument is that ZFS and jails provide the same functionality but are more stable than Docker. But as is it comes off as “get off my lawn you young whipper snappers”. But at the same time, the reason Docker won was not because it wa…

Docker was the first viable containerization technology on Linux. Despite the 15 year late start vs FreeBSD Jails, it's certainly winning by the numbers. But that has nothing to do with their respective UXs. It's a Linux vs FreeBSD signal.

> Docker was the first viable containerization technology on Linux.

No it wasn’t. Docker was late to the party even for Linux (and Linux was late compared to every other “UNIX”).

OpenVZ was around for years before docker. Its main issue was that it required out-of-tree kernel code. But there were some distributions that did still ship OpenVZ support. In fact it’s what Proxmox originally used. And it worked very well.

Then LXC came along. Fun fact, Docker was originally used LXC itself. But obviously that was a long time ago too.

I’ve used both OpenVZ and LXC in production systems before Docker came along. But I’ve always preferred FreeBSD Jails + ZFS to anything Linux has offered.

Re: Immutable Software Deploys Using ZFS Jails on FreeBSD

#45

Earlier quoted context omitted.

I never find arguments like this compelling (but I agree with your sentiment). I don’t much care to fire up the time machine to go back 12 or more years to develop software today. If your argument is that ZFS and jails provide the same functionality but are more stable than Docker. But as is it comes off as “get off my lawn you young whipper snappers”. But at the same time, the reason Docker won was not because it wa…

> especially when you are on Linux AND on macOS. I can’t run FreeBSD jails or ZFS on macOS, can I? Definitely not with one file and one command. On macOS, docker actually launches a Linux VM to run containers. If this counts, then yes, you can run FreeBSD jails or zfs on macOS, by running a FreeBSD VM.

But it works with one command and one (Docker)file. That’s what I mean by Docker being a kludgy solution: this is way less than ideal. But for developer experience this is very nice. And that same Dockerfile runs on everything from AWS, to GCP, to k8s, to Dokku, etc.

I dislike the implementation but I cannot deny that the UX is good enough to be very popular.

Re: Immutable Software Deploys Using ZFS Jails on FreeBSD

#47

I highly recommend this approach. I run ~all of my digital footprint on FreeBSD jails. After surveying how many jail managers have come and go in the last decade, I decided to just roll my own using a single shell script called jailctl [0]. Nothing fancy, just VNET jails based on ZFS templates (vanilla FreeBSD rootfs) and epair interfaces (which I truck to various VLANs on the host's egress interface). One pattern th…

Very neat!

Re: Immutable Software Deploys Using ZFS Jails on FreeBSD

#48
post #23

Earlier quoted context omitted.

I think there's FreeBSD images for all the clouds now. You would need to do more work yourself to fetch and run jails probably, and I don't know if there's a hosted repository of 'jail images', but in return, you'd probably have a nicer system (at least, I'd like such a system more than running containers on google container optimized linux)

Not Hetzner, sadly.

You can always upload your own, it's pretty simple doing so in a reproducible manner using something like Packer, but even without it you can just boot a VM into a rescue system, write your OS of choice to the VM disk and reboot.

Re: Immutable Software Deploys Using ZFS Jails on FreeBSD

#49
post #4

Earlier quoted context omitted.

No. ZFS has been stable in FreeBSD for something like 17 years, and FreeBSD jails have been around for something like 25 years. By the time Docker hit 1.0 (about 11 years ago), the use of snapshots and jails had already been normal parts of life in the FreeBSD space for over half of a decade.

I never find arguments like this compelling (but I agree with your sentiment). I don’t much care to fire up the time machine to go back 12 or more years to develop software today. If your argument is that ZFS and jails provide the same functionality but are more stable than Docker. But as is it comes off as “get off my lawn you young whipper snappers”. But at the same time, the reason Docker won was not because it wa…

MacOS runs containers spinning VMs. That’s why it’s slow (processing and filesystem I/O) compared to Linux.

Same thing could be done in FreeBSD. Someone needs to put in the work…

Re: Immutable Software Deploys Using ZFS Jails on FreeBSD

#50
post #4

Earlier quoted context omitted.

No. ZFS has been stable in FreeBSD for something like 17 years, and FreeBSD jails have been around for something like 25 years. By the time Docker hit 1.0 (about 11 years ago), the use of snapshots and jails had already been normal parts of life in the FreeBSD space for over half of a decade.

I never find arguments like this compelling (but I agree with your sentiment). I don’t much care to fire up the time machine to go back 12 or more years to develop software today. If your argument is that ZFS and jails provide the same functionality but are more stable than Docker. But as is it comes off as “get off my lawn you young whipper snappers”. But at the same time, the reason Docker won was not because it wa…

One of Docker's big advantages is its client/server architecture made it easier to run on all those macOS and Windows boxes by just spinning up a VM for the server side, then making that VM fully managed to where a large percent of users don't even know it's there.
Post reply on HN