Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why and how we’re migrating many of our servers from Linux to the BSDs

it-notes.dragas.net

Post reply on HN
Print view View original

11–20 of 237 posts

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#11

Post by codezero »

codezero wrote: post #10
cyberax wrote: post #9

> As an experiment, I decided to migrate two hosts (each with about 10 VMs) of a client — where I had full control—without telling them, over a weekend. Yeah. That guy should not be allowed anywhere near the production workloads. "I solve problems", my ass.

The client is paying for the VM. The underlying system is an abstraction. As long as service agreements weren’t interrupted I don’t see the problem. It sounds shady to say “without telling them,” because saying so implies they should have. I do a lot of optimizations for my customers without telling them, it’s not usually worth mentioning. I assume what they intended to convey was that this change caused no interrupt…

They also made it clear they tested it by deploying to a system not hosting clients VMs.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#13

Post by _huayra_ »

lotharcable wrote: post #8
Borg3 wrote: post #3

I wish he could write up a bit about XFS failure he had. Im using it from many many years and there is no issues at all.

It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…

Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to work well.

I don't know about it's multi-disk story (I do use ZFS for that personally), but for single disk options it is great. You get so many of the ZFS benefits (snapshots, rollback, easily create and delete volumes, etc) with MUCH lower memory usage (at least in my own experiments to try this out).

[0] https://lwn.net/Articles/824855/ [1] https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#15

Post by kiwijamo »

codezero wrote: post #10
cyberax wrote: post #9

> As an experiment, I decided to migrate two hosts (each with about 10 VMs) of a client — where I had full control—without telling them, over a weekend. Yeah. That guy should not be allowed anywhere near the production workloads. "I solve problems", my ass.

The client is paying for the VM. The underlying system is an abstraction. As long as service agreements weren’t interrupted I don’t see the problem. It sounds shady to say “without telling them,” because saying so implies they should have. I do a lot of optimizations for my customers without telling them, it’s not usually worth mentioning. I assume what they intended to convey was that this change caused no interrupt…

This is similar to AWS S3 object storage -- AWS has over the years changed how they store their S3 data -- however as long as the API responds the same way every time it's all good. Personally I would probably do some A/B testing -- migrate half the workload and compare A to B to see if the new system is performing better before migrating the other half.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#16

Post by gaadd33 »

_huayra_ wrote: post #13
lotharcable wrote: post #8

Earlier quoted context omitted.

It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…

Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…

Is the structure of 800gb btrfs containers mentioned in [0] how user data is stored? Just sharded across billions of containers?

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#18

Post by djbusby »

_huayra_ wrote: post #13
lotharcable wrote: post #8

Earlier quoted context omitted.

It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…

Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…

Why is this grey/down? Is there something factually incorrect?

Edit: it's less grey now.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#19

Post by cyberax »

codezero wrote: post #10
cyberax wrote: post #9

> As an experiment, I decided to migrate two hosts (each with about 10 VMs) of a client — where I had full control—without telling them, over a weekend. Yeah. That guy should not be allowed anywhere near the production workloads. "I solve problems", my ass.

The client is paying for the VM. The underlying system is an abstraction. As long as service agreements weren’t interrupted I don’t see the problem. It sounds shady to say “without telling them,” because saying so implies they should have. I do a lot of optimizations for my customers without telling them, it’s not usually worth mentioning. I assume what they intended to convey was that this change caused no interrupt…

> The client is paying for the VM. The underlying system is an abstraction.

The VM change was sufficient enough to alter the runtime of a task by several times. This is NOT a small inconsequential change.

You _have_ to warn your clients when you do stuff like this.

Re: Why and how we’re migrating many of our servers from Linux to the BSDs

#20

Post by cyberax »

kiwijamo wrote: post #15
codezero wrote: post #10

Earlier quoted context omitted.

The client is paying for the VM. The underlying system is an abstraction. As long as service agreements weren’t interrupted I don’t see the problem. It sounds shady to say “without telling them,” because saying so implies they should have. I do a lot of optimizations for my customers without telling them, it’s not usually worth mentioning. I assume what they intended to convey was that this change caused no interrupt…

This is similar to AWS S3 object storage -- AWS has over the years changed how they store their S3 data -- however as long as the API responds the same way every time it's all good. Personally I would probably do some A/B testing -- migrate half the workload and compare A to B to see if the new system is performing better before migrating the other half.

No, it's not. S3 has a very well defined API with easily measureable performance parameters. So AWS updates can make sure they don't make things worse.

This is not possible with a client's workload unless you can actually test it. That's why AWS will warn you multiple times if they need to migrate your EC2 instance onto a different hardware node. Even if it is technically "better".

Of course, the fact that clients trust their workloads to this guy probably means that there was nothing important there.

Post reply on HN