Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why and how we’re migrating many of our servers from Linux to the BSDs

it-notes.dragas.net

Post reply on HN
Print view View original

221–230 of 237 posts

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#221

Post by martinbaun »

stackskipton wrote: post #160
martinbaun wrote: post #152

Earlier quoted context omitted.

You're probably going to be downvoted to oblivion but I 100% agree with you. It seems like they tried to remove all the complexity, and just made new complexity.

Modern systems are complex generally. If it's not Kubernetes with YAML files, it's a bunch of servers in VMware which is own ball of wax followed by extensive Ansible/Puppet/Chef setup and SREs who keeping entire architecture in their head since they don't have time to write it down. Obviously, there is exact opposite with stuff like fly.io but they can be extremely constraining.

Yeah, I really liked Heroku and I used it for so many years, but it also killed itself kinda. It could never reduce prices as all machines "was the same" and thus a reduction in price meant lose of revenue and now they're just super expensive comparably.

Now I just dump Caddyserver and install things bare metals servers. I avoid all dependencies like the plague

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#222

Post by tiberious726 »

lproven wrote: post #134
thoroughburro wrote: post #130

> The largest failure was with btrfs — after a reboot, a 50 TB filesystem (in mirror, for backups) simply stopped working. No more mounting possible. Data was lost, but I had further backups. The client was informed and understood the situation. Within a few days, the server was rebuilt from scratch on FreeBSD with ZFS — since then, I haven’t lost a single bit. As someone who admins a lot of btrfs, it seems very unli…

> As someone who admins a lot of btrfs, it seems very unlikely that this was unrecoverable. As someone who used it all day every day in my day job for 4 years, I find it 100% believable. I am not saying you're wrong: I'm saying, experiences differ widely, and your patterns of use are not be universal. It's the single most unreliable untrustworthy filesystem I've used in the 21st century.

As someone who has used it in my day job since 2014, I find it around 5% believable. I've had nasty performance issues on old kernels, but never a single instance of unrecoverable data loss, and I've run it in plenty of pathological cases. Experiences differ

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#223

Post by tiberious726 »

electricant wrote: post #196
curt15 wrote: post #171

Earlier quoted context omitted.

Synology only uses btrfs in single-disk mode and implements RAID-1 functionality using its own patched version of mdadm to side-step the gotchas of native btrfs raid1.

What 'gotchas' exactly?

None for raid 1. They do it for raid 5/6 if you're a crazy person and want to run parity raid in 2024.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#224

Post by tiberious726 »

badsectoracula wrote: post #179
sidewndr46 wrote: post #145

Earlier quoted context omitted.

Why do people use btrfs and similar filesystems for production use? They are by no means dumpster fires. But the internet is littered with stories of "X happened, then I realized Y & that I wasn't getting my data back"

Btrfs has some nice features - e.g. compression and snapshots, which i didn't knew i'd even like before using them. Not only they have saved me a few times from bad updates ("saved" in the sense that i was able to pretty much instantly revert, it saved time, i wouldn't lose anything even without btrfs), but they also help with things like "i'm going to run this script to process 29837894293 files - and the script mig…

Nearly all of the stories of unrecoverable data loss I've read involve someone discovering they have a problem, then trying the traditional ext/xfs recovery techniques before reading the docs, and thereby destroying their fs.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#225

Post by tiberious726 »

traceroute66 wrote: post #178
SubjectToChange wrote: post #172

Earlier quoted context omitted.

Facebook (well, Meta, I guess) is famously a big user and developer of btrfs. It seems to work just fine for them.

> Facebook (well, Meta, I guess) is famously a big user and developer of btrfs. It seems to work just fine for them I really, really, really wish people would STOP with the whole "it works for $SilconValleyCorp so it must work for me" or "$SiliconValleyCorp does it, so I must". It only leads to disappointment in the case of the former and wholly un-necessary over-engineering in the case of the latter. (a) You do not…

They haven't fixed raid 5 because no one serious uses parity raid in this decade. Try recruiting an Open source dev to write something that useless...

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#226

Post by guilhas »

thoroughburro wrote: post #130

> The largest failure was with btrfs — after a reboot, a 50 TB filesystem (in mirror, for backups) simply stopped working. No more mounting possible. Data was lost, but I had further backups. The client was informed and understood the situation. Within a few days, the server was rebuilt from scratch on FreeBSD with ZFS — since then, I haven’t lost a single bit. As someone who admins a lot of btrfs, it seems very unli…

I was pleased with my home lab btrfs, had a 12TB raid1, and the PSU rail connected to the backplane sometimes would go down under load. Many scary errors but never lost anything. Took me 2 months to debug and replace the PSU

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#227

Post by __turbobrew__ »

homebrewer wrote: post #218
__turbobrew__ wrote: post #122

Earlier quoted context omitted.

Facebook has stacks of thousands of spare nodes ready at any moment to replace a failed node. All essential data will be replicated across many different boxes so if a box fails you just replace it with a fresh node and replicate the data there. This is much different to the consumer usecase where computers are pets and not cattle. A failed filesystem the night before you need to turn in your thesis may have a much l…

So much FUD in this discussion. Christ Mason talked publicly that they use the cheapest SSDs they can find (even worse things than what he would be willing to put in his laptop), and that they investigate every instance of btrfs corruption. You're saying the exact opposite of the main btrfs guy at Facebook. I wonder who is right...

Who is right, one guy whos reputation relies on something not breaking or a bunch of end users who report the thing broke for them?

I experienced issues with write amplification within the past few months in Ubuntu 22 so it isn’t like all the issues are gone. I do agree that there are less issues now than there was before, but I will still say that btrfs still breaks or behaves unexpectedly much more often than ext4 or xfs.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#228

Post by vilunov »

kev009 wrote: post #83
DeathArrow wrote: post #59

All is fine and dandy and BSDs can solve many use cases. Unfortunately for the solution we are working on, which imply many microservices we need Kubernetes and no BSD equivalent to Kubernetes exists.

I've helped build two top 10 service provider networks (10s of Tbps). One on FreeBSD, and one on Linux with Kubernetes. I don't really see Kubernetes as being a game changer. The biggest pro, it makes it easier to onboard both development and operations personnel having a quasi-standard for how a lot of things like scheduling and application networking work. But it also seems to come with a magnitude of accidental an…

Kubernetes is absolutely not an accidental complexity. Microservices are not an accidental complexity, and they are not a replacement for a proper repository.

Kubernetes solves administration of a cluster of Linux machines, as opposed to administering a single Linux machine. It abstracts away the concept of a machine, because it automates application scheduling, scaling across different machines, rolling updates of applications, adding/removing machines to the cluster all at the same time. There are no instruments like that for applications, the closest to them are something like Spark and Hadoop for data engineering tasks (not general applications).

Microservices are also used to solve a very specific problem – independent deployments of parts of the system. You can dance with your repository and your code directories as much as you want, if you're not in a very specific runtime (e.g. BEAM VM), you will not achieve independent deployments of parts of your service. The ability to "scale independently" (which tbh is mostly bullshit) is an accidental consequence of using HTTP RPC for microservice communication, which is also not the only way, but it allows reuse of the HTTP ecosystem.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#229

Post by MuffinFlavored »

cientifico wrote: post #184
MuffinFlavored wrote: post #170

Earlier quoted context omitted.

> it will probably expand your knowledge It really just fragments my knowledge to be honest. Say "I gotta get things done". Get me to a terminal. You've got Mac OS command line flags, GNU, BSD. Great. Then it's some kind of asinine config to interact with some piece of software, all to achieve "generally the same thing", just a different way/flavor. I really don't see the benefits.

It’s like driving a car. Most people can drive without needing to understand the internals, and it’ll still get them from point A to B. But just like some people enjoy diving deep into car mechanics, others enjoy understanding the intricacies of software. For me, that deeper knowledge is an advantage. It helps me quickly evaluate tradeoffs between databases, debug at the OS level, or dismiss a library still relying o…

I politely disagree. I love the driving a car analogy.

You've got 2 kinds of cars overall really. Automatic transmission or manual. Get into any of them and everything is the same. Gas pedal, brake pedal, etc.

Objective what you need to get done? Drive from A to B.

It is impossible to drive from A to B with FreeBSD if all you know is Linux commands/syntaxes. That's my whole argument. That it isn't as mild of a jump as driving car A or car B. You are blocked (and need to throw out what you know on Linux for BSD because it'll collide).

Are the differences mild? Sure, but enough to make you ineffective.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#230

Post by lproven »

homebrewer wrote: post #156
lproven wrote: post #136

Earlier quoted context omitted.

Me too. Repeatedly, at least once a year, on 3 different machines. The cause? Filling up the filesystem. Why? Because of OS snapshots. (Aside: why can they fill it? Because it doesn't give a straight answer to `df -h`. Why not? Because of snapshots.)

That happened recently? A few years ago they added a reserved area used for emergency purposes that should solve situations like that. Can't say I've run into these problems, although I don't tend to run btrfs very heavily because performance becomes unacceptable long before that due to CoW. https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html (Look for "GlobalReserve")

> That happened recently?

It happened to me repeatedly on both openSUSE Leap and openSUSE Tumbleweed during the 4 years I worked for SUSE: 2017-2021.

The `df` command doesn't work: it does not give reliable info. That alone disqualifies this FS for me.

The `fsck` equivalent does not work: every time I have tried, it corrupts volumes into unreadability.

Those 2 things are hard requirements for me.

I raised this internally as significant issues. They were dismissed.

Post reply on HN