Proposed server purchase for GitLab.com

Something that always seemed to cause nagging issues was that we wanted our cluster to have data encryption at rest. Ceph does not support this out of the box, which means that you need to use dmcrypt on top of your partitions, and present those encrypted partitions to Ceph. This requires some work to make sure that decrypt keys are setup properly, and that the machine can reboot automatically and remount the proper partitions. In addition, we ran into several issues where device mapper or otherwise would lock an OSD, which would send the entire machine into lockup, messy!

We also had to work pretty hard to build quality monitoring around Ceph - by default, there are very little tools that provide at-scale fine grained monitoring for the various components. We spent a lot of time figuring out what metrics we should be tracking, etc.

We also spent a good amount of time reaching out to other people and companies running ceph at scale to figure out how to tune and tweak it to work for us. The clusters were all-SSD, so there was a ton of work to tune the myriad of settings available, on ceph and the hosts themselves, to make sure we were getting the best possible performance out of the software.

When you run dozens-to-hundreds of servers with many SSDs in them that are doing constant traffic, you tend to hit every edge case in the hardware and software, and there are a lot of lurking demons. We went through controller upgrades, SSD firmware upgrades, tweaking OP settings, upgrading switches, finding that the write workload on certain SSDs caused problems and much more.

That's just a snapshot of some of the issues that we ran into with Ceph. It was a very fun project, but if you are getting into it for a high-throughput setup with hundreds of OSDs, it can be quite a bit of work.

Happy to chat more w/ anyone that's curious - there is some very interesting and fun stuff going on in the Ceph space.

We are running a 50%+ gross margin mid-stage venture-backed startup in Equinix facilities (but started there vs. cloud), and have no people near our facilities, and have had 0 issues service-wise related to doing management remotely. Yes, people go out to set up cabs, etc, but we hired our ops folks as generalists who had some network experience, and our CEO and CTO do as well, though AFAIK I don't have network logins active right now.

2 high-level thoughts I'd share:

1) Try not to use Ceph unless you're committed to having 2 people with deep experience at the code level.

2) I'd use Juniper QFX or EX, or Aristas. You don't seem to be running at scale or functionality where SDN magic is needed and there is a large community of QFC, EX, and Arista users your folks can reach out to when problems happen.

The other comments are more tuning and FYI on what we do HW-wise:

Specifically re: HW, at Kentik we run tens of worker nodes + flow ingest servers, all SM 1us w a few SSD and 256-384gb RAM. 48 logical cores, 2 x E5-2650v4.

We run approaching 1PB of storage, and while we still have some 4u 36-disk 3.5" boxes, those are phasing out and all we buy now is 2u SuperMicros w/ 24x2TB Samsung Evo 850ss. Procs are 72 logical core, 2 x E5-2697v4.

The Evo SSDs have been great - but our workload is largely appends or create/writes - largely but not all sequential, with high read IOPS. Before Samsung I was a big fan of Intel but we have no data on the modern Intels - slower for sure, but a focus on reliability is great...

We use JBOD and ZFS on the storage nodes; the LSI 9300-8i. Have things tested so we can do TRIM.

They do make SuperServers for roughly those configs, but we go with SM resellers who assemble and burn-in for +10-15%. I had 50+ SuperServers that were great at my Usenet company, but we'd rather have our ops folks work on things other than burn-in.

Happy to explain why we went to SSD vs. spinning at 2x the cost, but basically it made enough of a different at 95th and 99th percentile in our query times, and we had access to venture debt on great terms (which you should too and happy to discuss, since we're both funded by August).

Last note re: gear - when we were doing spinning, we found a screaming deal on new 2TB enterprise SATA (Hitachi, I think) for $50 and took the power/space hit for the +IOPS and extra compute we got for firing up the additional machines. Not sure if those are still out there, or the IOPS of this kind of approach would be needed.