Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

121–130 of 327 posts

Re: Proposed server purchase for GitLab.com

#121

Post by sytse »

dsr_ wrote: post #28

N1/N2: I think you are making a mistake by concentrating on the advantage of having a single node type. Databases really aren't like other systems. SuperMicro has a 4 node system available in which each node gets 6 disks, 2028-TP-xx-yyy. Get two of these, populate 2 nodes on each as your databases, and you can grow into the other spaces later. Run your databases on all-SSD; store backups on spinners elsewhere. Having…

Thanks, I agree that multiple nodes could be acceptable if needed.

So the proposal now is to have one chassis (excluding the backup system) and two types of nodes (normal and database).

Regarding the 2028 please see the conversation in https://news.ycombinator.com/item?id=13153853

Re: Proposed server purchase for GitLab.com

#122

Post by Spooky23 »

nik736 wrote: post #8

Are you sure about the location? I would go with Frankfurt, Germany. Biggest IX in the world and if you want a "low-latency" solution for all users this is basically the middle of everything. NYC will have a worse connection to Asia and I don't want to begin with India or something. While Frankfurt is basically only 70ms away from NY and around 120ms to the west coast, while even south america should be Just as a sug…

If the customer base has lots of US enterprise customers, that choice will cost you money. Ex-US data residency is an issue for many compliance standards.

Re: Proposed server purchase for GitLab.com

#123

Post by sytse »

peller wrote: post #35

For such a big expenditure, some prototyping first? Maybe buy a mainboard or two and some CPUs/HDDs/SSDs and benchmark them on your specific workloads. Also look into using something like bcache if going all-SSDs is too expensive.

We're in a rush because we need GitLab.com to get fast now. We might shoot ourselves in the foot by not prototyping first but we're taking the risk. We're hiring more consultants to help us with the move.

Bcache is mentioned in the article under the disk header.

Re: Proposed server purchase for GitLab.com

#124

Post by gr2020 »

gr2020 wrote: post #71

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the…

I just remembered a blog post I wrote a while back about exactly these points:

http://www.rassoc.com/gregr/weblog/2011/04/13/choosing-a-col...

Re: Proposed server purchase for GitLab.com

#125

Post by tomschlick »

I didn't see it mentioned but what are your plans for the network strategy. Are you planning to run dual-stack IPv4/IPv6 ? IPv4 only? Internal IPv6 only with NAT64 to the public stuff?

Hopefully IPv6 shows up somewhere in the stack. It's sad to see big players not using it yet.

Re: Proposed server purchase for GitLab.com

#126

Post by sytse »

nik736 wrote: post #38
mwcampbell wrote: post #37

I'm surprised to see that GitLab is using Unicorn. Isn't Unicorn grossly inefficient, because each of the worker processes can only handle one request at a time. Are web application processes actually CPU-bound these days? I don't know much about the Ruby web server landscape, but might Puma ( http://puma.io/ ) be better?

I was wondering the same. For my projects Puma has been awesome and better than Unicorn.

Puma is great. We tried it years ago and multithreading caused problems. This is likely due to problems in our application and its dependencies, not Puma. We reverted it in https://github.com/gitlabhq/gitlabhq/commit/3bc4845874112242...

Re: Proposed server purchase for GitLab.com

#127

Post by sytse »

bgwhn wrote: post #89

Regarding D2: If you're going to go with bcache, make sure you're using a kernel >= 4.5, since that's when a bunch of stability patches landed ( https://lkml.org/lkml/2015/12/5/38 ). Alternatively, if you're building your own kernel, you should be able to apply those patches yourself.

Thanks, added in https://gitlab.com/gitlab-com/www-gitlab-com/commit/da003427...

Re: Proposed server purchase for GitLab.com

#128

Post by alrs »

If the plan is still to build a huge Ceph-backed filesystem to store your git repos on, you are doomed.

Redhat, if you're out there: Now would be a good time to chime in about the limits of Ceph and the reasonable size of a filesystem.

Re: Proposed server purchase for GitLab.com

#129

Post by toomuchtodo »

KenoFischer wrote: post #109
sytse wrote: post #84

Earlier quoted context omitted.

Great points. We'll make sure to wire to separate power feeds that can both handle the entire load. Suggestions in how to calculate this? Taking the maximum rated load seems over the top.

I recently had to do this. The server I was putting up was rated for 3kW. To determine the expected load, I put it under a dummy load that I reasonably considered the maximum for what I would expect on the server (this was a dev machine, so I picked compiling the linux kernel as a benchmark). I ran that until the power stabilized (SuperMicro servers can measure power consumption in hardware and expose this via IPMI -…

Prime95! I haven't seen that since 2000-2001! Solid tool for burning in a box and putting the CPU under maximum load.

Re: Proposed server purchase for GitLab.com

#130

Post by sytse »

jscott0918 wrote: post #66

A bit confused why you are trying to roll your own storage solution. Ceph is great for a lot of applications, but I am not sure it really fits the bill for what you describe. Especially when you indicate you are going to use spinning disk behind it. Have you looked at any of the storage arrays on the market? Your TCO is likely to be much lower and your performance/resiliency much higher if you buy something with $100…

I think a storage appliance (NetApp, etc.) makes a lot of sense in the short term. The TCO is lower since we'll spend a lot of time making Ceph work.

In the longer term the storage appliance will lock us in and will get very expensive. I've heard pretty bad stories of companies betting on it. Especially with many small files like us (IOPS heavy).

And one goal of GitLab.com is to gain experience that we can reuse at our customers. Most of our customers use a storage appliance now but are interested in switching to something open source.

Post reply on HN