I may be way off base, but is this a solvable problem by just sharding/placing customer repositories over various storage systems?
Proposed server purchase for GitLab.com
191–200 of 327 posts
Re: Proposed server purchase for GitLab.com
#192Just a few quick notes. I've experience running ~300TB of usable Ceph storage. Stay away from the 8TB drives. Performance and recovery will both suck. 4TB drives still give the best cost per GB. Why are you using fat twins? Honestly, what does that buy you? You need more spindles, and fewer cores and memory. With your current configuration, what are you getting per rack unit? Consider a 2028u based system. 30 of thos…
Thanks for the great suggestions. We're considering the fat twins so we get both a lot of CPU and some disk. GitLab.com is pretty CPU heavy because of ruby and the CI runners that we might transfer in the future. So we wanted the maximum CPU per U. The 2028u has 2.5" drives. For that I only see 2TB drives on http://www.supermicro.com/support/resources/HDD.cfm for the SS2028U-TR4+. How do you suggest getting to 4TB?
Also whatever you do, don't buy all one kind of disk. That'll be the thing that dies first and most frequently. Buy from different manufacturers and through different vendors to try and get disks from at least a few different batches. That way you don't get hit by some batch of parts being out of spec by 5% instead of 2% and them all failing within a year, all at the same time.
If you do somehow manage to pick the perfect disk sure having everything from a single batch would be the best since that'll ensure you have the longest MTBF. But how sure are you that you'll be picking the perfect batch simply by blind luck?
Re: Proposed server purchase for GitLab.com
#193Earlier quoted context omitted.
I'm happy to see this - I could not agree more with these points. I think they are coming at this problem from the wrong perspective - instead of growing from virtual servers to their own dedicated hardware to get better CephFS performance, they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that t…
I don't see why it wouldn't be difficult to do a git implementation that replaces all the FS syscalls with calls to S3 or native ceph or some other object store. If all they're using NFS for is to store git, it seems like a big win to put in the up front engineering cost. I mean, especially because git's whole model of object storage is content-addressable and immutable, it looks like it's a prime use for generic obj…
Latency is an issue. Especially when traversing history in operations like log or blame, it is important to have an extremely low latency object store. Usually this means a local disk.
Re: Proposed server purchase for GitLab.com
#194I know Stackoverflow and all their related services are running bare-metal and has done so for a long time. They have written a very detailed series of blog posts about their whole infrastructure and hardware which I really recommend that you read if you haven't already.
Here's Part 1 of the series: http://nickcraver.com/blog/2016/02/17/stack-overflow-the-arc... (the other parts are linked from there)
Re: Proposed server purchase for GitLab.com
#195H1: I don't see a discussion of getting an ASN and doing BGP with a number of upstreams. You say you want a carrier-neutral facility, but that won't buy you much unless you have your own AS.
Some ISPs will let you announce on a /24, which shouldn't be too hard for GitLab to set up for themselves.
@sytse: If this (BGP + ASN + /24 IPv4 + /n IPv6) is your plan, I'd encourage you to get started on the process now.
Go ahead and apply for your ASN now and an IPv6 allocation. Then start working on the paperwork for an IPv4 allocation. Because there is no more IPv4 to allocate you'll have to go through the auction process and then the subsequent transfer process.
You'll easily be able to find a provider that can give you a /24 if you buy transit from them, but you don't wanna go through the trouble of renumbering into your own IP space if you can avoid it.
Re: Proposed server purchase for GitLab.com
#196[deleted]
Re: Proposed server purchase for GitLab.com
#197As someone already mentioned "all the other services" are missing (dns, ntp, monitoring, etc.), but also: - Shouldn't there be a puppet / chef / whatever deployment coordinator in there somewhere? - There's no mention of a virtualisation environment. While it's not a hardware issue really, all the extra services mentioned before will not take the whole server and you'll want to collocate some of them. (maybe even som…
- We would continue to use our existing cloud hosted Chef server for now. - We want to use Kubernetes instead of virtualization. - We provisioned 2 spare database servers for that reasons. - Git already compresses files with zlib so I'm not sure we can compress it much further https://git-scm.com/book/uz/v2/Git-Internals-Packfiles - The servers have a separate management port and "Apart from those routers we'll have…
I hope to god you are not using Kubernetes for your databases.
Re: Proposed server purchase for GitLab.com
#198Just a few quick notes. I've experience running ~300TB of usable Ceph storage. Stay away from the 8TB drives. Performance and recovery will both suck. 4TB drives still give the best cost per GB. Why are you using fat twins? Honestly, what does that buy you? You need more spindles, and fewer cores and memory. With your current configuration, what are you getting per rack unit? Consider a 2028u based system. 30 of thos…
Thanks for the great suggestions. We're considering the fat twins so we get both a lot of CPU and some disk. GitLab.com is pretty CPU heavy because of ruby and the CI runners that we might transfer in the future. So we wanted the maximum CPU per U. The 2028u has 2.5" drives. For that I only see 2TB drives on http://www.supermicro.com/support/resources/HDD.cfm for the SS2028U-TR4+. How do you suggest getting to 4TB?
Sorry, I was confused by the part numbers. I was thinking of the 6028u based system that have 12x3.5" drives. These are what I used for my OSD nodes in my Ceph deployment.
As for CPU density, I still feel like you're going to need more spindles to get the IO you're looking for.
Re: Proposed server purchase for GitLab.com
#199Earlier quoted context omitted.
Thanks for the great suggestions. We're considering the fat twins so we get both a lot of CPU and some disk. GitLab.com is pretty CPU heavy because of ruby and the CI runners that we might transfer in the future. So we wanted the maximum CPU per U. The 2028u has 2.5" drives. For that I only see 2TB drives on http://www.supermicro.com/support/resources/HDD.cfm for the SS2028U-TR4+. How do you suggest getting to 4TB?
Also whatever you do, don't buy all one kind of disk. That'll be the thing that dies first and most frequently. Buy from different manufacturers and through different vendors to try and get disks from at least a few different batches. That way you don't get hit by some batch of parts being out of spec by 5% instead of 2% and them all failing within a year, all at the same time. If you do somehow manage to pick the pe…
I had this problem with the Supermicro SATA DOMs. Had problems with the whole batch.
That said, I bought the same 6TB HGST disk for two years.
Re: Proposed server purchase for GitLab.com
#200Earlier quoted context omitted.
I'm curious to know if you think this work is within the core competency of GitLab. if so, how did you decide it was. If not, how do you realize the investment over time in something that isn't? Is the GitLab CEO here?
GitLab CEO here. Hardware and hosting are certainly not our core competencies. Hence all the questions in the blog post. And I'm sure we made some wrong assumptions on top of that. But it needs to become a core competency, so we're hiring https://about.gitlab.com/jobs/production-engineer/
I would challenge your assertion that you need a core competency in bare metal. AWS and GCE are performant enough--you're just not using them correctly. Invest in IaaS expertise and be successful; invest in bare metal at this day and age and live to regret it forever.