Live data from Hacker News

Why and how we’re migrating many of our servers from Linux to the BSDs

it-notes.dragas.net

211–220 of 237 posts

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#211

All I can say is experiences differ. I'm a long time Debian user, and now use FreeBSD for work. Both are far better than the proprietary competition, but I'd take Debian/Linux over FreeBSD when building a random server. To give but one example, I recently reported a bug when FreeBSD didn't boot after upgrade from 13 to 14. Worse the disk format was somehow altered so when the reboot tried to boot off 13 due to zfs bo…

> Who thought withering device names was a good idea, so that /dev no longer reflects the state of attached hardware? Sorry, could you clarify what this means? I'm not super familiar with freebsd and don't understand what withering means here.

Withering is the name the give to removing block devices under /dev that have file systems mounted on them. Contrast this to Linux, where /dev reflects something simpler - the hardware detected the kernel.

The /dev naming is about are how FreeBSD handles block device aliases. Like Linux, FreeBSD creates a number of aliases based on the block devices labels and uuid. My favourite Linux alias is missing on FreeBSD - bus path (which how you unambiguously get to the you just connected at a cable). On Linux these aliases are just symlinks to the real device, which means all it takes is "ls -l" to see the relationship between devices and aliases. Simple, elegant and it means all devices have one true name, so in error logs and so on you always know what device it's talking about.

Under FreeBSD these aliases are device nodes, so there is no single true name. The real device an alias maps to is not at all obvious. Worse, it's not the same device major or minor, and worse still the aliases behave differently. So for example, it the OS mounts as CDROM using it's label alias (which would be /dev/iso9660/label on FreeBSD) you can't eject it because the alias device doesn't understand the eject ioctl. But the you may not be able to get to it at all, because it's been withered away.

Complicating the issue still further is zfs. It wants to take over the roles of /etc/fstab and /sbin/mount. This gets particularly interesting when you boot off zfs, so there is no /dev, so there are no aliases, so it has no obvious way of figuring out what those path names you gave to zpool meant. They kludged their way around that somehow, but it doesn't always work - which I think is the trigger behind the boot failures I mentioned earlier. They took me days to figure out a work around. It was to turn off some of the aliasing.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#212
I dont think the article is even about BSDs, but generally really bad things in the tech sectors and philosophy to tech.

>my priority is solving my clients’ specific problems, not selling a predefined solution.

>It’s better to pay for everything to work than to pay to fix problems.

>computing should solve problems and provide opportunities to those who use it.

>The trend is to rush, to simplify deployments as much as possible, sweeping structural problems under the rug. The goal is to “innovate”, not necessarily improve — just as long as it’s “new” or “how everyone does it, nowadays.”

>Some people are used to thinking that the ideal solution is X — and believe that X is the only solution for their problems. Often, X is the hype of the moment

>When I ask, “Okay, but why? Who will manage it? Where will your data really be, and who will safeguard it?”, I get blank faces. They hadn’t considered these questions. No one had even mentioned them.

>We’ve lost control of the data. For many, it’s unnecessary to complicate things. And with every additional layer, we’re creating more problems.

Hopefully someday more people will wake up.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#213

Running 3 different BSDs is not my idea of solving problems.

They might be really good for specific tasks, but someone else will also need to maintain the setups, which will make things harder when most of the folks in the job market have experience with the various Linux distros, but finding BSD experience might be a tad more difficult.

I am personally on board with using the various BSDs when it makes sense (though maybe just pick FreeBSD and stick with it, as opposed to fragmenting the install base, the same way how I've settled on Ubuntu LTS wherever possible; it's not ideal but it works), except the thing is that most job ads and such call for Linux experience in particular, same with tooling like Kubernetes and OCI/Docker containers and such. Ergo, that's where most of my time goes, I want to remain employable and also produce solutions that will be understandable to most people, instead of getting my DMs pinged whenever someone is confused by what they're seeing.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#214
FreeBSD is great as a server. Wifi performance still sucks. The author praised byhve but byhve is not all it’s cracked up to be. Both Xen and Linux virtualization perform better, VMware as well. I like FreeBSD, but the other day I found it still uses sendmail. Rc.conf is simple to use, and the ports system is great…I just feel the author was pushing his “X” solution. Hardware support is important as well. I’ve used BSD for a SAN, and FW. Would I use it for a virtualization host? No.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#215

Earlier quoted context omitted.

Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…

Facebook has stacks of thousands of spare nodes ready at any moment to replace a failed node. All essential data will be replicated across many different boxes so if a box fails you just replace it with a fresh node and replicate the data there. This is much different to the consumer usecase where computers are pets and not cattle. A failed filesystem the night before you need to turn in your thesis may have a much l…

The ChromeOS Linux dev VM uses btrfs by the way.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#216
post #93

I really wanted to give a try to FreeBSD... because I thought it was linux from "better" times... and then I saw they are tied to the tons of similar gcc extensions... and then I say to myself "why bother since it has the same major compiler dependency issue", better try to fix linux code base or start from there.

It doesn’t, though? Gcc doesn’t even ship oob, the project itself uses clang, and almost all packages are also compiled with the system c/c++ toolchain.

llvm is not less worse than gcc... come on...

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#217

Earlier quoted context omitted.

Facebook (well, Meta, I guess) is famously a big user and developer of btrfs. It seems to work just fine for them.

> Facebook (well, Meta, I guess) is famously a big user and developer of btrfs. It seems to work just fine for them I really, really, really wish people would STOP with the whole "it works for $SilconValleyCorp so it must work for me" or "$SiliconValleyCorp does it, so I must". It only leads to disappointment in the case of the former and wholly un-necessary over-engineering in the case of the latter. (a) You do not…

> You do not know how or where Facebook use BTRFS

(S)he does, their employees explained it many times. They're very public about it.

> So they don't care if a random BTRFS instance borks itself.

They do, according to Christ Mason (IIRC) they investigate every instance of btrfs corruption, regardless of how unimportant the machine and data were. They're not any more frequent than with any other filesystem.

> Facebook probably employ the guy who invented BTRFS and an army of kernel developers on top of that

Not an "army" (only a few developers), but you're correct here.

> they STILL have not fixed RAID5 in BTRFS

Why would they? It's a niche technology that's only interesting to a few home users. I am a home user and have no use for it (or any of the alternatives like raidz).

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#218

Earlier quoted context omitted.

Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…

Facebook has stacks of thousands of spare nodes ready at any moment to replace a failed node. All essential data will be replicated across many different boxes so if a box fails you just replace it with a fresh node and replicate the data there. This is much different to the consumer usecase where computers are pets and not cattle. A failed filesystem the night before you need to turn in your thesis may have a much l…

So much FUD in this discussion. Christ Mason talked publicly that they use the cheapest SSDs they can find (even worse things than what he would be willing to put in his laptop), and that they investigate every instance of btrfs corruption. You're saying the exact opposite of the main btrfs guy at Facebook. I wonder who is right...

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#219

Earlier quoted context omitted.

> I always used PostgreSQL even though most were on MySQL That far back mysql and postgres were not even close to being a like-for-like comparison. One was a proper database, with things like referential integrity and a type system that didn't count the 31st of February to be a valid date, and the other was a glorified ndbm with some structure, a SQL interface, and was very very fast at running simple single-table SE…

Yet for most users none of that mattered. Because they would just use a SQL library or ORM which hid all of these details.

Correct. I said it was not a valid comparison if talking about proper databases, not that mysql was irredeemably bad and no use to anyone back then.

There were though a lot of people who probably should have used a better DB but used mysql through knowing no better or nothing else being available on cheap shared hosts. Many got lucky and got by but more than a few ended up running into problems or spending time implementing things (complete with bugs for later joy) in their BLL that really belong in the data layer. Similarly using ORMs away from their areas of core advantage is asking for problems later (though one of their core advantages _is_ to help with a quick turnaround on an MVP or other PoC, especially if you aren't much knowledgeable about DB design considerations at the time, so I can't criticise that much).

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#220

Earlier quoted context omitted.

Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…

Meta does a lot of things that don't scale for reliable/trustworthy systems and aren't suitable for all use-cases. (I also used to work there too.) ZFS is only reliable where it was battled-tested: on Solaris. ZoL has been infinitely tinkered with and smashed up that it's nothing like running a Thumper as a NAS. XFS + mdadm on Linux is, without a doubt, far more reliable than ZoL. Ask me how I know. I have the scars…

Yeah, my setup too, XFS + mdadm (+ eventually LVM2). Rock solid. It might not have HW raid performance, but in terms of stability, flexibility and recovery its absolutly unbeatable!
Post reply on HN