Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

131–140 of 327 posts

Re: Proposed server purchase for GitLab.com

#131

Post by sytse »

theptip wrote: post #74
sytse wrote: post #36

Earlier quoted context omitted.

We have sharding on the application layer in GitLab right now https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7273 and we're using it heavily to split the load among NFS servers. Then we have to think about redundancy. The simple solution is to have an secondary NFS server and use DRBD. For the shortcomings of that read http://githubengineering.com/introducing-dgit/ The next step is introducing more granular r…

That's a fair comment RE: outsourcing, but at my company I'd bias towards bringing some distributed computing knowledge in-house rather than bringing ops expertise plus maintenance burden in-house; sounds like you're going to have to add new expertise to your team either way. Worth investigating if you can bolt on a distributed datastore like etcd or ZooKeeper to store the cluster membership and data locations; this…

I agree we need to add new expertise to our team anyway. But I think adding bare metal expertise is easier than distributed system expertise.

Etcd is indeed very interesting. I'm thinking about using it for active active replication in https://gitlab.com/gitlab-org/gitlab-ee/issues/1381

Re: Proposed server purchase for GitLab.com

#132

Post by toomuchtodo »

tootie wrote: post #112
sytse wrote: post #97

Earlier quoted context omitted.

We looked at providers such as Softlayer but while they guarantee the performance of the servers they typically can't guaranty network latency. Since we're doing this to reduce latency https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ this is essential to us. We'll be glad to look into alternatives that manage the servers and network for us although the argument in https://news.ycombinator.com/item?id=131534…

None of that would convince me to lift a finger. Not unless you put dollar amounts on one vs the other. Version control isn't something I even care about performance.

Version control is the primary tool your developers are using at any org doing software development. It's performance is key to ensure devs aren't wasting their time waiting on version control.

Performance along with reliability are the most important metrics for someone providing VCS as a service.

Re: Proposed server purchase for GitLab.com

#133

Post by sytse »

Jabdoa wrote: post #87

Why go with 3.5' disks? You usually want 2.5' and you can fit 24 ones in 2HE. I recommend those using a supermicro cabinet. 2 cache SSDs and 20 SAS disks (+ 2 spares) in combination with a raid controller will give you very much quite fast storage at a sweet price spot.

What disk would you recommend for that setup?

Re: Proposed server purchase for GitLab.com

#134

Post by Spooky23 »

samstave wrote: post #20
sytse wrote: post #13

Earlier quoted context omitted.

Frankfurt surely has a great IX. Even if Frankfurt would make everyone better off (which I'm not sure about) there is another problem. People in the US are used to lower latencies because most SaaS services are hosted there.

I have less of a concern on latency than I do with who has taps in the lines... I would assume that the .de lines are saturated with 5 eyes...

Why would you care?

Re: Proposed server purchase for GitLab.com

#135

Post by sytse »

devicenull wrote: post #33

> We want to dual bound the network connections to increase performance and reliability. This will allow us to take routers out of service during low traffic times, for example to restart them after a software upgrade. does not really agree with > Each of the two physical network connections will connect to a different top of rack router. Sure, you can do it with something like MLAG, but that's really just moving you…

Thanks for the suggestions.

We're already planning a separate router for the management network ("Apart from those routers we'll have a separate router for a 1Gbps management network.").

All Ceph journals will be on SSD too. I've added a question about combining this with bcache in https://gitlab.com/gitlab-com/www-gitlab-com/commit/a9cc9aad...

Re: Proposed server purchase for GitLab.com

#137

Post by MichaelGG »

wmf wrote: post #83

I have built out a few racks of Supermicro twins. In general I would suggest hiring your ops people first and then letting them buy what they are comfortable with. C2: The Dell equivalent is C6320. CPU: Calculate the price/performance of the server , not the processor alone. This may lead you towards fewer nodes with 14-core or 18-core CPUs. Disk: I would use 2.5" PMR (there is a different chassis that gives 6x2.5" p…

Can you elaborate on what deficiencies 10GBase-T has in server applications?

Re: Proposed server purchase for GitLab.com

#138

Post by viraptor »

As someone already mentioned "all the other services" are missing (dns, ntp, monitoring, etc.), but also:

- Shouldn't there be a puppet / chef / whatever deployment coordinator in there somewhere?

- There's no mention of a virtualisation environment. While it's not a hardware issue really, all the extra services mentioned before will not take the whole server and you'll want to collocate some of them. (maybe even some of the main services too, if the resource usage on real hardware turns out wildly different than the estimates) If the choice is KVM, great. But if it's VMWare, you want to include that in the cost. (and the network model)

- The "staging x 1" is interesting... so what happens when you need to test a new version of postgres before deployment? You can't pretend that a test on one server (or 4 virtual ones) will be comparable to actual deployment, especially if you need to verify the real data performance.

- "Backing up 480TB of data [...] with a price of $0.01 per GB per month means that for $4800 we don't have to worry about much." - This makes a really bad assumption of X B of data produces X B of backup. That's a mirror, not a backup - it won't protect you from human errors if you overwrite your only mirror. For a backup you need to have actual system of restoring the data and a plan for how many past versions you want to store. On the other hand, gitlab data seems to be mostly text files - that should compress amazingly well, so that should also help with the restore speed.

- Network doesn't seem to mention any out of band access (iLO, or similar). That's another port per node required.

- Because tech is tech, I expect both the staging and spare servers to be repurposed soon to be used for "that one role we forgot about". One each is really not enough. (seen it happen so many times...)

Re: Proposed server purchase for GitLab.com

#139

Post by sytse »

lern_too_spel wrote: post #63

Since hosting git repositories is core to your business, you should take the time to do it right ( https://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/fi... ) instead of using vanilla git and relying on a magic filesystems and vertical scaling to solve your issues.

The presentation you linked is something we've considered. But it is build on top of Google's distributed filesystem. So we consider our move to Ceph a first step in that journey.

Re: Proposed server purchase for GitLab.com

#140

Post by aberoham »

pyrox420 wrote: post #100
sytse wrote: post #97

Earlier quoted context omitted.

We looked at providers such as Softlayer but while they guarantee the performance of the servers they typically can't guaranty network latency. Since we're doing this to reduce latency https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ this is essential to us. We'll be glad to look into alternatives that manage the servers and network for us although the argument in https://news.ycombinator.com/item?id=131534…

Been a softlayer customer for 4 years now. Their network is pretty awesome. When hosted in the same datacenter it's sub-ms response time always. If there is an issue they get right on it. You can even ask them to host the stuff in the same rack to get even better response time.

Same experience here, going on seven or eight years. They offer a great solution between cloud native and 100% owned/leased/caged bare metal.
Post reply on HN