Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why and how we’re migrating many of our servers from Linux to the BSDs

it-notes.dragas.net

Post reply on HN
Print view View original

141–150 of 237 posts

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#141

Post by wkat4242 »

hggigg wrote: post #98
kev009 wrote: post #83

Earlier quoted context omitted.

I've helped build two top 10 service provider networks (10s of Tbps). One on FreeBSD, and one on Linux with Kubernetes. I don't really see Kubernetes as being a game changer. The biggest pro, it makes it easier to onboard both development and operations personnel having a quasi-standard for how a lot of things like scheduling and application networking work. But it also seems to come with a magnitude of accidental an…

Agree with this on all points. We went from a virtualized server model to managed Kubernetes and costs have escalated considerably. The additional complexity and maintenance overheads of Kubernetes are not trivial and required additional staff hires just to keep things ticking. I think the cost so far from moving from two cages in separate datacentres running blades to AWS is approximately a 6x multiplier including s…

I think it mainly works for startups which can build it as a greenfield and are extremely sensitive to scaling (either they are exploding within 2 years or they're dead within 2 years). So they plan to scale from 0 to millions quickly. For that kinda business it makes sense because once the exploding growth comes it's super easy to scale up and even do that automatically. That's the power of Kubernetes.

For an existing business not in a hyper-growth phase with lots of restrictions and policies and processes and legacy stuff it's just extra work with no gain. It's just a round peg in a square hole. Wrong tool for the job.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#142

Post by baq »

threeseed wrote: post #117
hggigg wrote: post #98

Earlier quoted context omitted.

Agree with this on all points. We went from a virtualized server model to managed Kubernetes and costs have escalated considerably. The additional complexity and maintenance overheads of Kubernetes are not trivial and required additional staff hires just to keep things ticking. I think the cost so far from moving from two cages in separate datacentres running blades to AWS is approximately a 6x multiplier including s…

That's because you've moved to AWS which is an expensive cloud. You could have easily run Kubernetes on your virtualised servers.

IME nothing easy about maintaining your own storage cluster!

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#144

Post by nanolith »

nottorp wrote: post #143

If I want to run a BSD as just a file server - so I guess zfs + samba + bonjour or whatever the discovery protocol is these days - which BSD should I try?

I'd recommend using FreeBSD as your first BSD. It has a more recent version of ZFS integrated in the kernel than NetBSD. OpenBSD does not have ZFS support; it's a direction they chose not to take for security and simplicity reasons.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#145

Post by sidewndr46 »

thoroughburro wrote: post #130

> The largest failure was with btrfs — after a reboot, a 50 TB filesystem (in mirror, for backups) simply stopped working. No more mounting possible. Data was lost, but I had further backups. The client was informed and understood the situation. Within a few days, the server was rebuilt from scratch on FreeBSD with ZFS — since then, I haven’t lost a single bit. As someone who admins a lot of btrfs, it seems very unli…

Why do people use btrfs and similar filesystems for production use? They are by no means dumpster fires. But the internet is littered with stories of "X happened, then I realized Y & that I wasn't getting my data back"

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#146

Post by nucleardog »

graemep wrote: post #108
kev009 wrote: post #79

I use Linux, FreeBSD, NetBSD, and OpenBSD all for fun, learning, and profit(the first two). At the very least it is nice to make acquaintance with at least one BSD because it will probably expand your knowledge on Linux in ways you wont be able to anticipate. For example, FreeBSD got me into kernel development, full system debugging, network stack development, driver development, and understanding how the whole kit f…

> tend to use FreeBSD because in many ways it is simpler and less surprising, especially when accounting for the passage of time. ifconfig is still ifconfig, and it works great. rc.d is all I need for my own stuff. That sounds very appealing to me. I have to keep a small number of servers running, but its not my main focus and I would like to spend as little time on it as possible. I have started using Alpine Linux f…

The UI generally quite stable and well documented, which is awesome.

They also have things like `rpm` in ports that you can install. Why? Because you can enable linux binary compatibility[0] and run linux binaries on it (this implements the linux kernel interface, it's not a VM/emulator). It's also backward compatible with its own binaries back to FreeBSD 4 (circa 2000).

Though you may not need that as the ports/packages collection is pretty comprehensive.

It also comes with some nifty tools built-in for isolation (similar to but predating cgroups/containers) as "jails". It also has a hypervisor built in (bhyve) for virtualization if you do need to run any linux VMs or anything for any reason.

The way I usually sum up the difference to people is that FreeBSD is designed while Linux is grown. FreeBSD feels much more like a cohesive whole than Linux.

Really, the only reason I'm not running it everywhere is that the industry has kind of settled on linux-style containers for... absolutely everything, and the current solution for that on FreeBSD is basically "run linux in a VM".

[0] https://docs.freebsd.org/en/books/handbook/linuxemu/

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#147

Post by mijoharas »

rstuart4133 wrote: post #31

All I can say is experiences differ. I'm a long time Debian user, and now use FreeBSD for work. Both are far better than the proprietary competition, but I'd take Debian/Linux over FreeBSD when building a random server. To give but one example, I recently reported a bug when FreeBSD didn't boot after upgrade from 13 to 14. Worse the disk format was somehow altered so when the reboot tried to boot off 13 due to zfs bo…

> Who thought withering device names was a good idea, so that /dev no longer reflects the state of attached hardware?

Sorry, could you clarify what this means? I'm not super familiar with freebsd and don't understand what withering means here.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#148

Post by stackskipton »

taskforcegemini wrote: post #63
inopinatus wrote: post #41

Earlier quoted context omitted.

Sure, it's reasonable, if your desired outcome is to lose a customer by a) demonstrating a lack of burst or headroom capacity and/or graceful degradation/priority scheduling capability, and b) being a dick about it. Alternatively one may intervene with a positive approach and framing viz. "we have temporarily [provisioned additional capacity / relocated your storage to another spindle / mirrored your content to a les…

depends on what product/service plan they had. a 5$ per month is not worth jumping through the hoops, because usually an individual solution is needed, but this doesn't scale. if the site is important to the owner, they need a more fitting service plan with higher headroom. that being said, a good provider at least tries to filter out what could be a (d)dos

In 2010, those tools were not easily available outside massive Enterprise solutions like Akamai. Cloudflare just launched in 2009 and on-premise hardware was generally cost prohibitive.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#149

Post by yjftsjthsd-h »

blueflow wrote: post #95
INTPenis wrote: post #76

Earlier quoted context omitted.

Can't you just run k8s on bsd? You might have to build and maintain your own release of it, but I'm sure someone has done it already.

The BSD's don't have the namespaces/unshare syscall functionality like Linux does.

FreeBSD has jails, which AFAIK are fully a match in terms of functionality, but you would need to finish implementing containerd support (maybe via runj?) and then get k8s to run on top of that. I seriously doubt there's any actual technical blockers, it's just that nobody's finished implementing all the pieces and putting them together.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#150

Post by theamk »

> [users ..] began explicitly requesting “jails” instead of Docker hosts. They started using BastilleBSD to clone “template” jails and deploy them.

huh, were they running persistent docker containers and modifying them in-place? If that's the case, they were missing the best part of Docker - the Dockerfile and "container is a cattle". The power of Docker is there no ad-hoc system customization possible, it's all in Dockerfile which is source-controlled and versioned, and artifacts (like built images) are read-only.

To go from this to all-manual "use bastille edit TARGET fstab to manually update the jail mounts from 13.1 to 13.2 release path." [0] seems like a real step back. I can understand why one might want to go to BSD if they prefer this kind of workflow, but for all my projects, I am now convinced that functional-like approach (or at least IaaC-like one) is much more powerful than manually editing individual files on live hosts.

[0] https://bastille.readthedocs.io/en/latest/chapters/upgrading...

Post reply on HN