Proposed server purchase for GitLab.com
about.gitlab.com
Proposed server purchase for GitLab.com
1–10 of 327 posts
Re: Proposed server purchase for GitLab.com
#2I'll be here all day to learn from suggestions. I'm hoping for much feedback so please reference questions with the letter and number: 'Regarding R1'.
Re: Proposed server purchase for GitLab.com
#3Love how they did some number crunching and decided that rent vs own, own won. I think that if more places looked they would find that out also. There must be a margin in it since the big players are making money at it.
I'm interested to see what they end up with in the end.
Re: Proposed server purchase for GitLab.com
#4If you are looking for performance, do not get the 8TB drives. In my experience, drives above 5TB do not have good response times. I don't have hard numbers, but I built a 10 disk RAID6 array with 5TB disks and 2TB disks and the 2TB disks were a lot more responsive.
From my own personal experience, I would go with a PCIe SSD cache/write buffer, and then a primary SSD tier and a HDD tier. Storage, as it seems you guys have experienced, is something you want to get right on the first try.
Edit:
N1: Yes, the database servers should be on physically separate enclosures.
D5: Don't add the complexity of PXE booting.
For Network, you need a network engineer.
This is the kind of thing you need to sitdown with a VAR for. Find a datacenter and see who they recommend.
Re: Proposed server purchase for GitLab.com
#5Love how they did some number crunching and decided that rent vs own, own won. I think that if more places looked they would find that out also. There must be a margin in it since the big players are making money at it. I'm interested to see what they end up with in the end.
Thanks! The decision to move to metal was because of performance problems https://about.gitlab.com/2016/11/10/why-choose-bare-metal/
It is nice that we'll save on costs but we anticipate a lot of extra complexity that will slow us down. So if it wasn't needed we would have stayed in the cloud. But it is interesting that both our competitors (GitHub.com and BitBucket.org) also moved to metal.
Re: Proposed server purchase for GitLab.com
#6C1: The SuperMicro offers blade-server densities without the ridiculous prices that proprietary blade-server chassis come with. It's a pretty good system. Please make sure you have assessed your power requirements correctly and communicated that to your datacenter.
If you were larger, Open Compute Platform might be a way to go. Maybe next generation.
Re: Proposed server purchase for GitLab.com
#7Yeah, link aggregation doesn't work how they think it does. And not having a separate network for Ceph is going to bite them in the arse.
GitLab is fine software but fuck me, they need to hire someone with actual ops experience (based on this post and their previous "we tried running a clustered file system in the cloud and for some reason it ran like shit" post).
Re: Proposed server purchase for GitLab.com
#8Are you sure about the location? I would go with Frankfurt, Germany. Biggest IX in the world and if you want a "low-latency" solution for all users this is basically the middle of everything. NYC will have a worse connection to Asia and I don't want to begin with India or something. While Frankfurt is basically only 70ms away from NY and around 120ms to the west coast, while even south america should be Just as a suggestion since you seem to be so certain about the location. I basically know every DC in Frankfurt, so if you need any help or info in that regard feel free to contact me :-)
Re: Proposed server purchase for GitLab.com
#9If you are looking for performance, do not get the 8TB drives. In my experience, drives above 5TB do not have good response times. I don't have hard numbers, but I built a 10 disk RAID6 array with 5TB disks and 2TB disks and the 2TB disks were a lot more responsive. From my own personal experience, I would go with a PCIe SSD cache/write buffer, and then a primary SSD tier and a HDD tier. Storage, as it seems you guys…
Thanks for commenting! You're right that 2TB disks will have more heads per TB and therefore more IO. We want to increase performance of the 8TB drives with Bcache. If we go to smaller drives we won't be able to fit enough capacity in our common server that has only 3 drives per node. In this case we'll have to go to dedicated file servers, reducing the commonality of our setup. We're using JBOD with Ceph instead of RAID.
If the 8TB drives don't offer the IO we need we'll have to leave part of them empty. I assume that if you fill them with 2TB they should perform equally well. Word on the street is that Google offers Nearline to fill all the capacity they can't use because of latency restrictions for their own applications.
It is an interesting idea to do SSD cache + SSD storage + HDD instead of just SSD cache + HDD. I'm not sure if Ceph allows for this.
Re: Proposed server purchase for GitLab.com
#10I'll be here all day to learn from suggestions. I'm hoping for much feedback so please reference questions with the letter and number: 'Regarding R1'.
I'm curious to know if you think this work is within the core competency of GitLab. if so, how did you decide it was. If not, how do you realize the investment over time in something that isn't? Is the GitLab CEO here?