Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FreeBSD Bhyve Virtualization

vermaden.wordpress.com

Post reply on HN
Print view View original

11–20 of 30 posts

Re: FreeBSD Bhyve Virtualization

#11

Post by terminalcommand »

replete wrote: post #8

I check every now and then to see if there is MacOS guest support, I would really like to use BSD as a host and completely drop Apple hardware since they started soldering SSDs. I have the last Intel MBP, and sure ARM is powerful, but in 5 years of heavy use it will be a brick in need of a new mainboard and that makes no sense to me. I guess the reality is in a year or two I'll land on fedora and proxmox or something…

I just researched a bit, mac os x guest vm with pcie passthrough seems possible on linux.

Dropping the links below:

https://github.com/kholia/OSX-KVM

https://github.com/yoonsikp/macOS-KVM-PCI-Passthrough

Re: FreeBSD Bhyve Virtualization

#12

Post by replete »

sofixa wrote: post #10
replete wrote: post #8

I check every now and then to see if there is MacOS guest support, I would really like to use BSD as a host and completely drop Apple hardware since they started soldering SSDs. I have the last Intel MBP, and sure ARM is powerful, but in 5 years of heavy use it will be a brick in need of a new mainboard and that makes no sense to me. I guess the reality is in a year or two I'll land on fedora and proxmox or something…

Why do you need macOS guest support? IIRC none of the official reasons (like the forced developing/releasing of App Store apps only from macOS) would work in a non-supported virtualised environment due to Apple shenanigans with the hardware and software checks thereof.

I prefer MacOS as a daily driving operating system to linux or Windows for development and general computing.

Re: FreeBSD Bhyve Virtualization

#13

Post by replete »

terminalcommand wrote: post #11
replete wrote: post #8

I check every now and then to see if there is MacOS guest support, I would really like to use BSD as a host and completely drop Apple hardware since they started soldering SSDs. I have the last Intel MBP, and sure ARM is powerful, but in 5 years of heavy use it will be a brick in need of a new mainboard and that makes no sense to me. I guess the reality is in a year or two I'll land on fedora and proxmox or something…

I just researched a bit, mac os x guest vm with pcie passthrough seems possible on linux. Dropping the links below: https://github.com/kholia/OSX-KVM https://github.com/yoonsikp/macOS-KVM-PCI-Passthrough

I've been looking at this for years, there are options if you have suitable hardware (there are a number of caveats). Proxmox has looked the most promising for a while.

Re: FreeBSD Bhyve Virtualization

#14

Post by voytec »

I've recently switched from Proxmox to bhyve while upgrading my dedicated box lease. I use it for prod mail&web and various lab work. Should have done it earlier, as I don't really need the things bhyve lacks (suspend, HA/clustering) for my use-cases.

    # vm list
    NAME         DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
    access       default    bhyveload  2    8G      -    No       Stopped
    cicd         default    bhyveload  2    6G      -    No       Running (33065)
    feeds        default    grub       2    4G      -    No       Running (35227)
    gate         default    bhyveload  2    4G      -    Yes [1]  Running (21012)
    k8s-master0  default    grub       2    4G      -    Yes [4]  Running (1543)
    k8s-node0    default    grub       6    64G     -    Yes [5]  Running (9988)
    mail         default    bhyveload  3    8G      -    Yes [3]  Running (87082)
    ml           default    grub       4    24G     -    No       Running (64236)
    registry     default    bhyveload  2    2G      -    No       Running (47712)
    repos        default    bhyveload  2    2G      -    No       Running (14767)
    web          default    bhyveload  2    4G      -    Yes [2]  Running (92365)
Other than that, I use bhyve on my laptop daily since around 2015-2016. It was somewhat painful at first. I had to bake a CD key into Windows ISO for headless install but now VNC support exists and it's easy to output any graphical installer via VNC.

bhyve doesn't offer API and has not the most user-friendly interface (vm-bhyve[1] for the rescue!) but overall, I couldn't be happier with its - typical for FreeBSD - set-and-forget stability.

[1] https://github.com/churchers/vm-bhyve

Re: FreeBSD Bhyve Virtualization

#15

Post by magicalhippo »

replete wrote: post #8

I check every now and then to see if there is MacOS guest support, I would really like to use BSD as a host and completely drop Apple hardware since they started soldering SSDs. I have the last Intel MBP, and sure ARM is powerful, but in 5 years of heavy use it will be a brick in need of a new mainboard and that makes no sense to me. I guess the reality is in a year or two I'll land on fedora and proxmox or something…

I'm using Bhyve and doing PCIe passthrough of some NVMe's to a Linux guest.

To my slight surprise, PCIe passthrough was device dependent, ie not all PCIe devices were supported. I'm running TrueNAS which runs a slightly older FreeBSD kernel, and from what I could gather GPU passthrough would not work there. IIRC the latest FreeBSD has some support for that.

But network cards should work, and NVMe's definitely work.

However it wasn't straight sailing. Using the approach from the wiki[1] did not work directly, I had to mask the devices using a pre-init boot script with

    devctl set driver -f nvme1 ppt
[1]: https://wiki.freebsd.org/bhyve/pci_passthru

Re: FreeBSD Bhyve Virtualization

#16

Post by vermaden »

jonhermansen wrote: post #3

Hey vermaden, great article! I saw you had some problems with virt-manager, and I did too! There is a solution your problem: > I was able to start FreeBSD 13.2 installation … but it got frozen at the kernel messages and nothing more happened. If you notice at the bootloader screen, it says something like "5. Cons: Dual (Serial primary)" Pressing the 5 key at the bootloader screen will toggle the output from "Serial p…

Thank You - I will check that out.

If it will work out I will update the article and maybe even probably add another dedicated virt-manager/libvirt Bhyve article.

Regards, vermaden

Re: FreeBSD Bhyve Virtualization

#17

Post by mad_vill »

notthetup wrote: post #4

There was a bunch of discussion about the live migration features of bhyve on a recent episode of “Oxide and Friends” https://m.youtube.com/watch?v=eQR98smFYTc

It’s funny this article popped up. I have been trying to get the oxide vmm working on freebsd in my spare time.

Too bad I don’t know nearly enough about rust, bhyve, openbsd or illumos to ever come close to get this working.

Re: FreeBSD Bhyve Virtualization

#18

Post by crest »

vermaden wrote: post #16
jonhermansen wrote: post #3

Hey vermaden, great article! I saw you had some problems with virt-manager, and I did too! There is a solution your problem: > I was able to start FreeBSD 13.2 installation … but it got frozen at the kernel messages and nothing more happened. If you notice at the bootloader screen, it says something like "5. Cons: Dual (Serial primary)" Pressing the 5 key at the bootloader screen will toggle the output from "Serial p…

Thank You - I will check that out. If it will work out I will update the article and maybe even probably add another dedicated virt-manager/libvirt Bhyve article. Regards, vermaden

For FreeBSD VMs I would recommend removing the VNC console. If the only console is a TTY FreeBSD does the right thing out of the box. It also requires a lot less bandwidth to tunnel a serial console than a video stream of a text console.

Re: FreeBSD Bhyve Virtualization

#19

Post by crest »

moondev wrote: post #5
ggm wrote: post #2

The only thing missing is migration from old iohyve saved zfs images to vm-bhyve or libvirt Otherwise, a super and very complete guide. Well worth bookmarking if you want to do virtualised os on FreeBSD.

Nested virtualization?

The problem with nested virtualisation is that it isn't fully hardware accelerated. The missing features (shadow paging, etc.) have to emulated in software. Such emulation requires lots of delicate error prone code.

Re: FreeBSD Bhyve Virtualization

#20

Post by vermaden »

ggm wrote: post #2

The only thing missing is migration from old iohyve saved zfs images to vm-bhyve or libvirt Otherwise, a super and very complete guide. Well worth bookmarking if you want to do virtualised os on FreeBSD.

From what I checked the iohyve puts disk images as ZFS zvol volumes like:

- /dev/zvol/zroot/iohyve/VMNAME/disk0

So the migration would be:

- create your new VM with disk of the same size

- check your new VM disk place at 'vm-bhyve':

    # vm info VM | grep system-path 
    system-path: /vm/VM/disk0.img
- copy contents of 'iohyve' disk into VM disk

    # dd bs=1m if=/dev/zvol/zroot/iohyve/VMNAME/disk0 of=/vm/VM/disk0.img
- start your new VM

This should work.

Same with libvirt/virt-manager - just copy your VM image from /var/lib/libvirt/images/ dir and it should work the same.

Hope that helps.

Hope that helps.

Post reply on HN