Proposed server purchase for GitLab.com

I think they are coming at this problem from the wrong perspective - instead of growing from virtual servers to their own dedicated hardware to get better CephFS performance, they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that they can expose over NFS. At some point in the future, it will bite them. Not an if, but a when.

In addition, this means that running physical hardware, CephFS and Kubernetes (among other things) are now going to be part of their core competencies - I think they are going to underestimate the cost to GitLab in the long run. When they need to pay for someone to be a 30 minute drive from their DC 24/7/365 after the first outage, when they realize how much spare hardware they are going to want around, etc.

As someone who has run large scale Ceph before (though not CephFS, thankfully), it's not easy to run at scale. We had a team of 5 software engineers as well as an Ops and Hardware team, and we had to do a lot to get it stable. It's not as easy as installing it and walking away.

In a long run, probably. Immediately after deploying, unless they hire very experienced, I expect quite a few "never seen before" issues (may not result in publicly visible downtime though). Monitoring for "thermal events", weird and hard to debug issues requiring firmware updates, "bad cable" issues, etc. are not what you have to deal with in the cloud.

GitLab is awesome. I'm really sad that in the past two or three months I've only found one GitLab link on HN to click. There really needs to be more. (I'm not sure if this is because I'm browsing in AEDT or if GitLab isn't used a lot on here.)

I wondered about how you guys might do advertising to get more mindshare, and then I realized one possible explanation about why you're doing this: getting technical advice from the community means everyone's had a part to play, and they're likely to remember that. Good move ;)

---

In my case, I have little (okay, 0) practical experience; a lot of the following is mentioned experimentally, to see how these ideas would handle the described environment. It's pretty much all stuff I've read online.

I welcome replies that shoot down any of these ideas.

> Disk

Disks can be slow so we looked at improving latency. Higher RPM hard drives typically come in GB instead of TB sizes. Going all SSD is too expensive. To improve latency we plan to fit every server with an SSD card. On the fileservers this will be used as a cache. We're thinking about using Bcache for this.

There's already been another brief comment (https://news.ycombinator.com/item?id=13153317) about ZFS.

So, I'll ask. Why not ZFS? You don't have to run FreeBSD anymore to get a stable implementation.

You can put both the L2ARC and ZIL on SSDs. You can even use striping with them. Don't quote me on this but I think there MAY be some recovery capabilities built into these layers for if the power goes out (either it didn't use to be possible and now it is, or it's architecturally impossible, I hilariously cannot remember which).

---

> In general 1GB of memory per TB of raw ZFS disk space is recommended.

This is ONLY if you have dedupe switched on. If you have dedupe off you can run systems in just 4GB. A lot of home server enthusiasts do this.

There are a lot of unfortunate and widespread misconceptions about ZFS.

---

(This bit's somewhat anecdotal and is more informational than actionable. It's worth noting if you're interested in disks.)

> Every node can fit 3 larger (3.5") harddrives. We plan to purchase the largest one available, a 8TB Seagate with 6Gb/s SATA and 7.2K RPM.

Technically, the largest one available (on Amazon and presumably elsewhere) right now is 10TB, but its price/capacity ratio is atrocious compared to the rest of the market ($450-$520 per disk).

I've heard that Seagate Enterprise Capacity drives either die within the first 2-4 weeks or last 20 years. They have 5 year warranties in any case. I haven't heard anything else about other disks.

Very interestingly, 8TB seems to be the current market leader. Here are a bunch of prices I took straight off Amazon, as guides:

#3: 4TB: $170 (13 disks for 52TB = $2040)

#2: 5TB: $200 (10 disks for 50TB = $2000)

#4: 6TB: $239 (9 disks for 54TB = $2151)

#1: 8TB: $360 (7 disks for 56TB = $1673)

#5: 10TB: $450 (5 disks for 50TB = $2250)

A little while ago 5TB was the leader, and I was going to argue for more disks.

---

(Hitting add comment now instead of waiting so I can keep up with the discussion)

(I'm from Germany, but I couldn't care less about the country per se or GitLab being co-located in Frankfurt)