Live data from Hacker News

Why and how we’re migrating many of our servers from Linux to the BSDs

it-notes.dragas.net

31–40 of 237 posts

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#31
All I can say is experiences differ. I'm a long time Debian user, and now use FreeBSD for work. Both are far better than the proprietary competition, but I'd take Debian/Linux over FreeBSD when building a random server.

To give but one example, I recently reported a bug when FreeBSD didn't boot after upgrade from 13 to 14. Worse the disk format was somehow altered so when the reboot tried to boot off 13 due to zfs bootonce flag (supposedly a failsafe), it refused to boot for the same reason. I believe it's due to a race condition in geom/cam. The same symptoms were reported 6 years ago, but the bug report has seen no activity. Making your system irrecoverable without a rescue image and console access strikes me as pretty serious. He waxes lyrical about zfs, but it's slower and more resource hungry than it's simpler competition and it's not difficult to find numerous serious zfs bug reports over the years. (But not slower than FreeBSD's UFS, oddly. It's impressively slow.) Another thing that sticks in my mind is a core zfs contributor saying it's encryption support should never have been merged.

This sounds too disparaging because the simplicity and size of FreeBSD has its own charms, but the "it's all sunshine and roses" picture he paints doesn't ring true to me. While it's probably fair to say stable versions of FreeBSD are better than the Linux kernels from kernel.org, and possibly Fedora and Ubuntu, they definitely trail behind the standard Debian stable releases.

Comparing FreeBSD to Debian makes throws up some interesting tradeoffs. On the one hand, FreeBSD's init system is a delight compared to systemd. Sure, systemd can do far, far more. But that added complexity makes for a steep learning curve and a lot of weird and difficult to debug problems, and as FreeBSD's drop dead simple /etc/rc.conf system proves most of the time the complexity isn't needed to get the job done. FreeBSD's jails just make more intuitive sense to me than Linux's equivalent which is built on control groups. FreeBSD's source is a joy to read compared to most I've seen elsewhere. I don't know who's responsible for that - but they deserve a medal.

On the downside - what were they thinking when they came up with the naming scheme under /dev for block devices? (Who thought withering device names was a good idea, so that /dev no longer reflects the state of attached hardware?) And a piece of free advice - just copy how Linux has does it's booting. Loading a kernel + initramfs is both simpler and far more flexible then the FreeBSD loader scheme. Hell, it's so flexible you can replace a BIOS with it.

The combination of the best parts Linux and the BSD's would make for an wonderful world. But having a healthy selection of choices is probably more important, and yes - I agree with him that if you are building an appliance that has an OS embedded in it, the simplicity of FreeBSD does give it an edge.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#33
post #3

I wish he could write up a bit about XFS failure he had. Im using it from many many years and there is no issues at all.

It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…

"One of the reasons Linux persists despite not having something up to the level of ZFS is that most of ZFS features are redundant to larger enterprise customers."

ZFS is used heavily on Linux and runs well, though there are some limitations which are being addressed over time in the OpenZFS project. It is used across all areas that Linux serves, whether laptop, desktop, home server all the way to enterprise. https://openzfs.org/wiki/Main_Page

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#34

One of the website providers I use (Pair) for 20+ years used to be exclusively FreeBSD. I believe they use a lot of Linux now. Not sure why.

I used them for shared personal website hosting until the Obama era when they were mostly/all FreeBSD. I moved off around the time AWS came about.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#36

I switched from FreeBSD to Linux, mainly because of the bad Java support and the simple fact that Linux became way more popular, which added to the difference in software availability.

FreeBSD has pretty good Java support? Sure if there is a new LTS release it takes a couple of months before it is ported to FreeBSD but that's about it?

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#37

Earlier quoted context omitted.

It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…

Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…

Meta does a lot of things that don't scale for reliable/trustworthy systems and aren't suitable for all use-cases. (I also used to work there too.)

ZFS is only reliable where it was battled-tested: on Solaris. ZoL has been infinitely tinkered with and smashed up that it's nothing like running a Thumper as a NAS.

XFS + mdadm on Linux is, without a doubt, far more reliable than ZoL. Ask me how I know. I have the scars to prove it.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#38

Earlier quoted context omitted.

It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…

"One of the reasons Linux persists despite not having something up to the level of ZFS is that most of ZFS features are redundant to larger enterprise customers." ZFS is used heavily on Linux and runs well, though there are some limitations which are being addressed over time in the OpenZFS project. It is used across all areas that Linux serves, whether laptop, desktop, home server all the way to enterprise. https://…

ZFS on Linux is not really usable for most users because every kernel update can break your ZFS compatibility.

Meaning unless you want to put in the time to manually test every kernel update and ensure your kernel version stays in-sync with OpenZFS you can very likely end up with an unbootable system.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#39

One of the website providers I use (Pair) for 20+ years used to be exclusively FreeBSD. I believe they use a lot of Linux now. Not sure why.

Ah, pair. Reminds me of this gem: http://www.archub.org/ycoutage.txt

Seems reasonable to me:

... This site and this directory were driving up server load averages, which causes instability for all users on this server. Our block on this site appears to have blocked over 150,000 connections in about half an hour. This is simply more than is appropriate for a shared server.

You may want to review your domain's traffic logs to see what kind of traffic this site is getting. The logs for today will be distributed at midnight EST tonight. If the traffic to this site is legitimate, you should look into a dedicated or a virtual private server. If the traffic is not legitimate, you should block it. I can provide assistance with that if you need it.

This domain and this directory will need to be disabled until traffic dies down, traffic is blocked, or your account is upgraded from shared hosting. If you need access to the directory so that you can make any necessary changes, please let me know.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#40

Earlier quoted context omitted.

Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…

Meta does a lot of things that don't scale for reliable/trustworthy systems and aren't suitable for all use-cases. (I also used to work there too.) ZFS is only reliable where it was battled-tested: on Solaris. ZoL has been infinitely tinkered with and smashed up that it's nothing like running a Thumper as a NAS. XFS + mdadm on Linux is, without a doubt, far more reliable than ZoL. Ask me how I know. I have the scars…

ZFS has been plenty battle tested on FreeBSD.
Post reply on HN