Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why and how we’re migrating many of our servers from Linux to the BSDs

it-notes.dragas.net

Post reply on HN
Print view View original

121–130 of 237 posts

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#121

Post by [deleted] »

jsiepkes wrote: post #38
shrubble wrote: post #33

Earlier quoted context omitted.

"One of the reasons Linux persists despite not having something up to the level of ZFS is that most of ZFS features are redundant to larger enterprise customers." ZFS is used heavily on Linux and runs well, though there are some limitations which are being addressed over time in the OpenZFS project. It is used across all areas that Linux serves, whether laptop, desktop, home server all the way to enterprise. https://…

ZFS on Linux is not really usable for most users because every kernel update can break your ZFS compatibility. Meaning unless you want to put in the time to manually test every kernel update and ensure your kernel version stays in-sync with OpenZFS you can very likely end up with an unbootable system.

[deleted]

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#122

Post by __turbobrew__ »

_huayra_ wrote: post #13
lotharcable wrote: post #8

Earlier quoted context omitted.

It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…

Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…

Facebook has stacks of thousands of spare nodes ready at any moment to replace a failed node. All essential data will be replicated across many different boxes so if a box fails you just replace it with a fresh node and replicate the data there.

This is much different to the consumer usecase where computers are pets and not cattle. A failed filesystem the night before you need to turn in your thesis may have a much larger impact on your life.

Another thing to consider is that Facebook runs btrfs on enterprise hardware (including SSDs with battery backups) which is going to be much more reliable than some chromebook which lives in the bottom of your backpack that you bring on transit every day.

Finally, I will say that the copy on write features of btrfs can result is some wildly different behaviour based upon how you use it. You can get into some very bad pathological cases with write amplification, and if you run btrfs on top of LUKS it can nearly be impossible to figure out why your disk is being pegged despite very little throughput at the VFS layer.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#123

Post by quesera »

vfclists wrote: post #51
jdbernard wrote: post #43

Earlier quoted context omitted.

> I would work for myself, following my own philosophy. Sounds like he understood it just fine. He owns the means of production.

He only works for himself only if he gets a steady income stream so long as the systems he manages stay up, which I'm sure he doesn't. There is a reason for the "you will own nothing and you will be happy" ideology being promoted by "PTB", ie subscriptions for ink cartridges, heated seats and advanced suspensions in newer cars. Corporations now want a continuous income stream from the services provide by physical pro…

> There is a reason for the "you will own nothing and you will be happy" ideology being promoted by "PTB",

No one, and certainly not the "PTB", is promoting this idea.

It was a thought experiment in a book/TED talk that deeply offends a certain percentage of the populace, and is ignored by everyone else. It is nothing more than that.

It does not exist in any discussions outside the circles of those who are offended for profit, and those that listen to them.

Spare yourself (and us), and stop listening to them.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#124

Post by quesera »

Tor3 wrote: post #88
vfclists wrote: post #51

Earlier quoted context omitted.

He only works for himself only if he gets a steady income stream so long as the systems he manages stay up, which I'm sure he doesn't. There is a reason for the "you will own nothing and you will be happy" ideology being promoted by "PTB", ie subscriptions for ink cartridges, heated seats and advanced suspensions in newer cars. Corporations now want a continuous income stream from the services provide by physical pro…

I have kind of scratched my head a bit as to why our local VM provider has all these scheduled downtimes.. while the servers we run in-house have those maybe once or twice a year, at most. Hm..

> why our local VM provider has all these scheduled downtimes

Generally (but not exclusively) this is because they have more complex infrastructure, a larger number of customers, and a contractual commitment to give advance notice of outages when possible.

It therefore makes sense to schedule downtime even if you don't need it (or even use it), so that customers can arrange their own expectations.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#125

Post by krageon »

threeseed wrote: post #115
kev009 wrote: post #83

Earlier quoted context omitted.

I've helped build two top 10 service provider networks (10s of Tbps). One on FreeBSD, and one on Linux with Kubernetes. I don't really see Kubernetes as being a game changer. The biggest pro, it makes it easier to onboard both development and operations personnel having a quasi-standard for how a lot of things like scheduling and application networking work. But it also seems to come with a magnitude of accidental an…

Kubernetes has unquestionably been a game changer. If you look at most enterprises today you will see it deployed everywhere. And most of the complexity has been abstracted away by the cloud providers so all you're left with is a system that can handle all manner of different applications and deployment scenarios in a consistent way.

> Kubernetes has unquestionably been a game changer.

> If you look at most enterprises today you will see it deployed everywhere.

This doesn't mean it's a game changer. It just means it has a big cargo-cult and people keep using it regardless of whether they need it.

> most of the complexity has been abstracted away by the cloud providers

The abstractions that people add make things more complex, not less. Unless of course you don't care to understand what you're running, which is precisely the problem with the culture around most folks that use kubernetes (and more broadly the US cloud providers as an entity).

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#126

Post by sitkack »

kev009 wrote: post #79

I use Linux, FreeBSD, NetBSD, and OpenBSD all for fun, learning, and profit(the first two). At the very least it is nice to make acquaintance with at least one BSD because it will probably expand your knowledge on Linux in ways you wont be able to anticipate. For example, FreeBSD got me into kernel development, full system debugging, network stack development, driver development, and understanding how the whole kit f…

I really really enjoyed using FreeBSD. It isn't a distro but a full OS that you can and easily fully rebuild from source. The people are great, the tech is good.

The CON is coming up, https://freebsdfoundation.org/news-and-events/event-calendar...

Fall 2024 FreeBSD Summit November 7-8, 2024 San Jose, CA

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#127

Post by viraptor »

draga79 wrote: post #109
viraptor wrote: post #104

Earlier quoted context omitted.

It's still something that's weird to do without notifying the customers. What if things were slower? What if bsd introduced some slight change in behaviour that messed up their data but they didn't know when/why things changed? Full control doesn't mean unexpected YOLO changes are welcome.

As specified in several parts, the tests were conducted while maintaining and using BSD-based infrastructures for over 20 years. In some cases, Linux was used for various reasons (commercial, ideological, because they were inherited infrastructures managed by others, etc.), but the results were anticipated. I did not expect a performance degradation, and in any case, having set up the systems in a mirrored environmen…

Maybe it was all done properly. I hope so. But the post really doesn't show that which I guess is what a few people here notice. You can't say you tested something for today for over 20 years. Things change - if it wasn't tested for that specific migration for that specific customer, then it wasn't really tested. I see people doing yolo changes that way and thought it's worth mentioning explicitly.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#129

Post by bravetraveler »

_huayra_ wrote: post #13
lotharcable wrote: post #8

Earlier quoted context omitted.

It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…

Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…

> I don't know about it's multi-disk story (I do use ZFS for that personally), but for single disk options it is great.

I can reliably, across vendors and drives, break RAID10 on BTRFS where MD+LVM are totally fine. Simply pull power. Discovered this when building out my latest workstation.

I haven't tried other configurations; after finding this pattern I decided to leave BTRFS for single-disk configurations where I want CoW

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#130

Post by thoroughburro »

> The largest failure was with btrfs — after a reboot, a 50 TB filesystem (in mirror, for backups) simply stopped working. No more mounting possible. Data was lost, but I had further backups. The client was informed and understood the situation. Within a few days, the server was rebuilt from scratch on FreeBSD with ZFS — since then, I haven’t lost a single bit.

As someone who admins a lot of btrfs, it seems very unlikely that this was unrecoverable. btrfs gets itself into scary situations, but also gets itself out again with a little effort.

In this instance “I solve problems” meant “I blow away the problem and start fresh”. Always easier! Glad the client was so understanding.

Post reply on HN