I may be way off base, but is this a solvable problem by just sharding/placing customer repositories over various storage systems?
Proposed server purchase for GitLab.com
81–90 of 327 posts
Re: Proposed server purchase for GitLab.com
#82Love how they did some number crunching and decided that rent vs own, own won. I think that if more places looked they would find that out also. There must be a margin in it since the big players are making money at it. I'm interested to see what they end up with in the end.
Thanks! The decision to move to metal was because of performance problems https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ It is nice that we'll save on costs but we anticipate a lot of extra complexity that will slow us down. So if it wasn't needed we would have stayed in the cloud. But it is interesting that both our competitors (GitHub.com and BitBucket.org) also moved to metal.
Would love to chat with you to see if a switch to GCP might solve your performance and pricing issues. It's also always interesting to see how GCP vs AWS vs bare metal fairs.
email me at bookman@google.com and I'll be sure to get you in contact with the right people at google.
Re: Proposed server purchase for GitLab.com
#83I have built out a few racks of Supermicro twins. In general I would suggest hiring your ops people first and then letting them buy what they are comfortable with.
C2: The Dell equivalent is C6320.
CPU: Calculate the price/performance of the server, not the processor alone. This may lead you towards fewer nodes with 14-core or 18-core CPUs.
Disk: I would use 2.5" PMR (there is a different chassis that gives 6x2.5" per node) to get more spindles/TB, but it is more expensive.
Memory: A different server (e.g. FC630) would give you 24 slots instead of 16. 24x32GB is 768GB and still affordable.
Network: I would not use 10GBase-T since it's designed for desktop use. I suggest ideally 25G SFP28 (AOC-MH25G-m2S2TM) but 10G SFP+ (AOC-MTG-i4S) is OK. The speed and type of the switch needs to match the NIC (you linked to an SFP+ switch that isn't compatible with your proposed 10GBase-T NICs).
N1: A pair of 128-port switches (e.g. SSE-C3632S or SN2700) is going to be better than three 48-port. Cumulus is a good choice if you are more familiar with Linux than Cisco. Be sure to buy the Cumulus training if your people aren't already trained.
N2: MLAG sucks, but the alternatives are probably worse.
N4: No one agrees on what SDN is, so... mu.
N5: SSE-G3648B if you want to stick with Supermicro. The Arctica 4804IP-RMP is probably cheaper.
Hosting: This rack is a great ball of fire. Verify that the data center can handle the power and heat density you are proposing.
Re: Proposed server purchase for GitLab.com
#84Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the…
fully agreed here, especially leave yourself room for when you misjudged some resource utilization (and need more). Nothing worse than having a resource crunch (cpu/mem/io) and not being able to resolve it because your rack is out of power/cooling/etc - you'll come to appreciate how easy it was in cloud just clicking the button and turning out your wallet.
Great points. We'll make sure to wire to separate power feeds that can both handle the entire load. Suggestions in how to calculate this? Taking the maximum rated load seems over the top.
Re: Proposed server purchase for GitLab.com
#85I'm surprised to see that GitLab is using Unicorn. Isn't Unicorn grossly inefficient, because each of the worker processes can only handle one request at a time. Are web application processes actually CPU-bound these days? I don't know much about the Ruby web server landscape, but might Puma ( http://puma.io/ ) be better?
Puma is something we want to eventually run, but right now it's not really a priority. Instead we're focusing on response timings and memory usage. See the following links for more information:
Re: Proposed server purchase for GitLab.com
#86Earlier quoted context omitted.
It doesn't work like that. 2TB drive will always be faster than an 8TB drive. The amount of data has no effect when compared to the physical attributes of the drive. More platters will increase the response time. Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier.
> Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier. By "Tiering", is this moving data between different drive types? Or by moving the data to different parts of the platter to optimize for rotational physics?
By Tiering, I'm talking about moving blocks of data from slower to faster or visa verse. If I have 10TB of 10k IOPS storage and 100TB of 1k IOPS storage in a tiered setup, data that is frequently accessed would reside in the 10k IOPS tier while less frequently accessed data would be in the 1k tier. In this case, the blocks of popular repositories would be stored in SSD, while the blocks of your side project that you haven't touched in 4 years would be on the HDD. You still have access to it, it might just take a bit longer to clone.
Ceph can probably explain it better than I can. http://docs.ceph.com/docs/jewel/rados/operations/cache-tieri...
Re: Proposed server purchase for GitLab.com
#87Why go with 3.5' disks? You usually want 2.5' and you can fit 24 ones in 2HE. I recommend those using a supermicro cabinet. 2 cache SSDs and 20 SAS disks (+ 2 spares) in combination with a raid controller will give you very much quite fast storage at a sweet price spot.
Re: Proposed server purchase for GitLab.com
#88[deleted]
Re: Proposed server purchase for GitLab.com
#89Regarding D2: If you're going to go with bcache, make sure you're using a kernel >= 4.5, since that's when a bunch of stability patches landed (https://lkml.org/lkml/2015/12/5/38). Alternatively, if you're building your own kernel, you should be able to apply those patches yourself.
Re: Proposed server purchase for GitLab.com
#90I have built out a few racks of Supermicro twins. In general I would suggest hiring your ops people first and then letting them buy what they are comfortable with. C2: The Dell equivalent is C6320. CPU: Calculate the price/performance of the server , not the processor alone. This may lead you towards fewer nodes with 14-core or 18-core CPUs. Disk: I would use 2.5" PMR (there is a different chassis that gives 6x2.5" p…
Wow Wes, these are all awesome suggestions. All of them (CPU, disk, memory, network, hosting) are things we'll consider doing. I added the Dell server with https://gitlab.com/gitlab-com/www-gitlab-com/commit/34bd78d8...
Would you mind if we contact you to discuss?