Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FreeBSD Bhyve Virtualization

vermaden.wordpress.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: FreeBSD Bhyve Virtualization

#21

Post by pmooney »

mad_vill wrote: post #17
notthetup wrote: post #4

There was a bunch of discussion about the live migration features of bhyve on a recent episode of “Oxide and Friends” https://m.youtube.com/watch?v=eQR98smFYTc

It’s funny this article popped up. I have been trying to get the oxide vmm working on freebsd in my spare time. Too bad I don’t know nearly enough about rust, bhyve, openbsd or illumos to ever come close to get this working.

While illumos carries a port of bhyve which is largely similar to upstream FreeBSD, there are several areas where it diverges. Propolis (the userspace VMM component) relies on some of those differences to function, especially when it comes to live migration.

Re: FreeBSD Bhyve Virtualization

#22

Post by DannyBee »

I seem to remember zvol having known performance issues that were improved greatly in the latest openzfs release.

If you read https://github.com/openzfs/zfs/issues/11407

It should be in there somewhere.

Dataset plus raw file will almost always be faster though due to the rest of the issues mentioned

Re: FreeBSD Bhyve Virtualization

#23

Post by mad_vill »

pmooney wrote: post #21
mad_vill wrote: post #17

Earlier quoted context omitted.

It’s funny this article popped up. I have been trying to get the oxide vmm working on freebsd in my spare time. Too bad I don’t know nearly enough about rust, bhyve, openbsd or illumos to ever come close to get this working.

While illumos carries a port of bhyve which is largely similar to upstream FreeBSD, there are several areas where it diverges. Propolis (the userspace VMM component) relies on some of those differences to function, especially when it comes to live migration.

Well I notice that alias from the propolis repo :).

Thanks for the heads up. Will probably ditch this effort.

Really want to play around with some of the oxide stuffs but it’s hard with illumos not supporting nested virt.

Re: FreeBSD Bhyve Virtualization

#24

Post by doublerabbit »

bHyve in jails is a god send.

I can create a jail, pass bHyve to it, allocate resources limits, assign a virtual nic enabling the jail to utilise firewalls. And then hand the jail IP to a client. Enabling them to create as many virtual machines their resources allows.

Using Jails this itself enables another level of security.

As if a VM hack breaks free, itself is then only isolated to the jail. And, if I was to be truely paranoid I could create a fortress jail with sub jails with a jail in the sub jail allowing bHyve to operate within.

Backups are only a matter of backing up the jail. ZFS does this without sweat.

Re: FreeBSD Bhyve Virtualization

#25

Post by wkat4242 »

sofixa wrote: post #10
replete wrote: post #8

I check every now and then to see if there is MacOS guest support, I would really like to use BSD as a host and completely drop Apple hardware since they started soldering SSDs. I have the last Intel MBP, and sure ARM is powerful, but in 5 years of heavy use it will be a brick in need of a new mainboard and that makes no sense to me. I guess the reality is in a year or two I'll land on fedora and proxmox or something…

Why do you need macOS guest support? IIRC none of the official reasons (like the forced developing/releasing of App Store apps only from macOS) would work in a non-supported virtualised environment due to Apple shenanigans with the hardware and software checks thereof.

Huh? When I still virtualized macOS everything worked just fine. I think the last one was 11 or 12. ESXi has now dropped support and Mac has not much relevance for me anymore.

I heard the apple silicon platform is a bit tighter but that can't really be virtualized on x86.

Re: FreeBSD Bhyve Virtualization

#26

Post by ilyt »

voytec wrote: post #14

I've recently switched from Proxmox to bhyve while upgrading my dedicated box lease. I use it for prod mail&web and various lab work. Should have done it earlier, as I don't really need the things bhyve lacks (suspend, HA/clustering) for my use-cases. # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE access default bhyveload 2 8G - No Stopped cicd default bhyveload 2 6G - No Running (33065) feeds default grub…

You could just use plain libvirt/KVM, virt-manager works well enough and can be used to manage remote libvirt hypervisors too.

In previous company we pretty much went from Proxmox to bare tools once we got the hang of it.

Re: FreeBSD Bhyve Virtualization

#27

Post by ilyt »

doublerabbit wrote: post #24

bHyve in jails is a god send. I can create a jail, pass bHyve to it, allocate resources limits, assign a virtual nic enabling the jail to utilise firewalls. And then hand the jail IP to a client. Enabling them to create as many virtual machines their resources allows. Using Jails this itself enables another level of security. As if a VM hack breaks free, itself is then only isolated to the jail. And, if I was to be t…

That's interesting way to do shared hosting, can jails limit by CPU time ?

Re: FreeBSD Bhyve Virtualization

#28

Post by doublerabbit »

ilyt wrote: post #27
doublerabbit wrote: post #24

bHyve in jails is a god send. I can create a jail, pass bHyve to it, allocate resources limits, assign a virtual nic enabling the jail to utilise firewalls. And then hand the jail IP to a client. Enabling them to create as many virtual machines their resources allows. Using Jails this itself enables another level of security. As if a VM hack breaks free, itself is then only isolated to the jail. And, if I was to be t…

That's interesting way to do shared hosting, can jails limit by CPU time ?

You can with rctl [0][1].

Another hack was one where you could use tcsh shell and /etc/login launching an application as a man in the middle to limit and launch a process with cpu limits.

[0] https://klarasystems.com/articles/controlling-resource-limit...

[1] https://wiki.freebsd.org/Hierarchical_Resource_Limits

Re: FreeBSD Bhyve Virtualization

#29

Post by voytec »

ilyt wrote: post #26
voytec wrote: post #14

I've recently switched from Proxmox to bhyve while upgrading my dedicated box lease. I use it for prod mail&web and various lab work. Should have done it earlier, as I don't really need the things bhyve lacks (suspend, HA/clustering) for my use-cases. # vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE access default bhyveload 2 8G - No Stopped cicd default bhyveload 2 6G - No Running (33065) feeds default grub…

You could just use plain libvirt/KVM, virt-manager works well enough and can be used to manage remote libvirt hypervisors too. In previous company we pretty much went from Proxmox to bare tools once we got the hang of it.

I'm using a system that I know and a hypervisor that I come to trust. Previous setup had different requirements.

Re: FreeBSD Bhyve Virtualization

#30

Post by vermaden »

jonhermansen wrote: post #3

Hey vermaden, great article! I saw you had some problems with virt-manager, and I did too! There is a solution your problem: > I was able to start FreeBSD 13.2 installation … but it got frozen at the kernel messages and nothing more happened. If you notice at the bootloader screen, it says something like "5. Cons: Dual (Serial primary)" Pressing the 5 key at the bootloader screen will toggle the output from "Serial p…

I tried it and the Serial console was greyed out - I could not select it.

Probably when I would force that upon the FreeBSD loader(8) maybe it will work - but there are other issues.

The Network Type 6 is not supported - so I was not able to create a NAT.

I could start a VM without any networks - but that would render then unusable for me.

I will try again in the future when the libvirt(8) driver for Bhyve is better implemented and more complete.

Regards, vermaden

Post reply on HN