I switched from FreeBSD to Linux, mainly because of the bad Java support and the simple fact that Linux became way more popular, which added to the difference in software availability.
Why and how we’re migrating many of our servers from Linux to the BSDs
21–30 of 237 posts
Re: Why and how we’re migrating many of our servers from Linux to the BSDs
#22[dead]
Re: Why and how we’re migrating many of our servers from Linux to the BSDs
#23Earlier quoted context omitted.
It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…
Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…
I am stuck in the btrfs-is-buggy mindset precisely because it managed to lose my root partition on a single disk machine. It might also have raid problems, but not exclusively.
Re: Why and how we’re migrating many of our servers from Linux to the BSDs
#24Earlier quoted context omitted.
The client is paying for the VM. The underlying system is an abstraction. As long as service agreements weren’t interrupted I don’t see the problem. It sounds shady to say “without telling them,” because saying so implies they should have. I do a lot of optimizations for my customers without telling them, it’s not usually worth mentioning. I assume what they intended to convey was that this change caused no interrupt…
> The client is paying for the VM. The underlying system is an abstraction. The VM change was sufficient enough to alter the runtime of a task by several times. This is NOT a small inconsequential change. You _have_ to warn your clients when you do stuff like this.
The workload got several times faster. The customer’s only concern was that they might be accidentally running on a more expensive instance.
In every system I’ve worked on, the agreement is in terms of an SLO. We never gave our customers any sort of expectation (or guarantee) that we wouldn’t suddenly wildly beat our SLO targets (and, in fact, we often did, due to routine upgrades).
Having said that, certain customers dictate production freezes during launches, or only want to run stuff that’s been baked in production elsewhere for 3-6 months. Upgrading those customers behind their backs would be unacceptable, especially because they pay extra for a crappier but more stable setup.
Re: Why and how we’re migrating many of our servers from Linux to the BSDs
#25Earlier quoted context omitted.
This is similar to AWS S3 object storage -- AWS has over the years changed how they store their S3 data -- however as long as the API responds the same way every time it's all good. Personally I would probably do some A/B testing -- migrate half the workload and compare A to B to see if the new system is performing better before migrating the other half.
No, it's not. S3 has a very well defined API with easily measureable performance parameters. So AWS updates can make sure they don't make things worse. This is not possible with a client's workload unless you can actually test it. That's why AWS will warn you multiple times if they need to migrate your EC2 instance onto a different hardware node. Even if it is technically "better". Of course, the fact that clients tr…
The author also converted some of these VMs to jails, so I assume they have root on the VMs (and the customers want them to admin the host). That means they should be able to see the application level performance metrics.
Re: Why and how we’re migrating many of our servers from Linux to the BSDs
#26Earlier quoted context omitted.
> The client is paying for the VM. The underlying system is an abstraction. The VM change was sufficient enough to alter the runtime of a task by several times. This is NOT a small inconsequential change. You _have_ to warn your clients when you do stuff like this.
The workload got several times faster. The customer’s only concern was that they might be accidentally running on a more expensive instance. In every system I’ve worked on, the agreement is in terms of an SLO. We never gave our customers any sort of expectation (or guarantee) that we wouldn’t suddenly wildly beat our SLO targets (and, in fact, we often did, due to routine upgrades). Having said that, certain customer…
If your customer panics at you over something you did, you might have done goofed.
Re: Why and how we’re migrating many of our servers from Linux to the BSDs
#27Earlier quoted context omitted.
It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…
Facebook runs their entire stack using Btrfs [0]. I would encourage anyone who is stuck in the "oh btrfs is so buggy and loses data" mindset (not helped by articles like this [1] that play off btrfs as some half-baked contraption, when it's really btrfs raid that needs a LOT more time to bake) to look into things and realize that large companies (OpenSuse, Redhat, Faceboook) have poured a lot of time to get it to wor…
> Facebook runs their entire stack using Btrfs
Yeah, and when I was there machines would run out of disk space at 50% usage and it took months to figure out why. In the mean time, they'd just reimage the machine and hope. I don't recall any issues with data loss, but it didn't have the air of reliability.
But my team was weird at FB, our uptimes of 45 days were way above the average, and we ran into all sorts of things because we operated outside the norm.
Re: Why and how we’re migrating many of our servers from Linux to the BSDs
#28[flagged]
Please don't do this here. Not that we don't appreciate a good old-fashioned flame, but that the long-term costs outweigh the benefits, and we want this site to survive long-term.
Re: Why and how we’re migrating many of our servers from Linux to the BSDs
#29the title of the talk is "Why (and how) we’re migrating many of our servers from Linux to the BSDs" and that should be the title of this post too. I like that the blog post shares the slides, not just the video.
Ok, done. Thanks!
Re: Why and how we’re migrating many of our servers from Linux to the BSDs
#30I wish he could write up a bit about XFS failure he had. Im using it from many many years and there is no issues at all.
It hard to know the timeline with his data loss, but I am assuming it was a long time ago. XFS is originally from SGI Irix and was designed to run on higher end hardware. SGI donated it to Linux in 1999 and it carried a lot of its assumptions over. For example on SGI boxes you had "hardware raid" with cache, which essentially is a sort of embedded computer with it's own memory. That cache had a battery backup so that…
> I understand FreeBSD uses UFS that uses a "WAL" or "write ahead log".. where it records writes it is going to do before it does it.
I think you're describing UFS soft-updates? I think that's more or less for meta data updates, not data data. It's been a while since I reviewed it, but it gets you nice things like snapshots and background fsck so after an unclean restart your system can get back to work immediately and clean up behind the scenes. There is some sort of journalling that's fairly new, but my experience from 10 years ago was soft-updates and background fsck just worked; and if you wanted better, ZFS was probably what you want, if you can afford copy on write.