Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

211–220 of 327 posts

Re: Proposed server purchase for GitLab.com

#211

Post by daxelrod »

manacit wrote: post #144
Spooky23 wrote: post #114

I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…

I'm happy to see this - I could not agree more with these points. I think they are coming at this problem from the wrong perspective - instead of growing from virtual servers to their own dedicated hardware to get better CephFS performance, they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that t…

> they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that they can expose over NFS

As someone who administers GitLab for my company, yes please.

Any high availability scenario that involves "just mount the same NFS volume on your standby" is a nonstarter for us. (We've found mounting NFS across different datacenters to be too unreliable and our failover scenarios include loss of a data center.)

It would also be wonderful to be able to run GitLab in any of the PaaSes that only have ephemeral disk, but that's a secondary concern.

Re: Proposed server purchase for GitLab.com

#212

Post by late2part »

kijiki wrote: post #169
MichaelGG wrote: post #137

Earlier quoted context omitted.

Can you elaborate on what deficiencies 10GBase-T has in server applications?

Latency, power and cost. The PHY has to do a lot of forward error correction and filtering, so it adds latency (for the FEC), power (for all the DSP) and cost (for the silicon area to do all of the above).

The latency is almost certainly immaterial, 0.3uSec v. 2uSec. That's MICROseconds not MILLIseconds. Power draw used to be an issue, but not anymore.

Consider the power pull from copper v fiber listed here [2]. The Arista 7050TX 128 port pulls 507W while the 7056SX 128 port pulls 235W. Yes, copper is more but we're talking half a kW for 2 TOR switches. And for this you get much cheaper cabling, as the SFP+ are much more expensive (go AOC if you do go fiber, BTW) and you have to worry about clean fiber, etc.

Where fiber is REALLY nice is with the density, although the 28AWG [3] almost makes that moot.

There's few DC builds that can't do end-to-end copper any more, at least for the initial 50 racks.

[1] - http://www.datacenterknowledge.com/archives/2012/11/27/data-... [2] - https://www.arista.com/en/products/7050x-series [3] - https://www.monoprice.com/product?p_id=13510

Re: Proposed server purchase for GitLab.com

#213

Post by devicenull »

sytse wrote: post #135
devicenull wrote: post #33

> We want to dual bound the network connections to increase performance and reliability. This will allow us to take routers out of service during low traffic times, for example to restart them after a software upgrade. does not really agree with > Each of the two physical network connections will connect to a different top of rack router. Sure, you can do it with something like MLAG, but that's really just moving you…

Thanks for the suggestions. We're already planning a separate router for the management network ("Apart from those routers we'll have a separate router for a 1Gbps management network."). All Ceph journals will be on SSD too. I've added a question about combining this with bcache in https://gitlab.com/gitlab-com/www-gitlab-com/commit/a9cc9aad...

If you're doing this right, the management network will not actually be accessible via the normal operating system. Most IPMI controllers support sharing a normal nic and management (meaning both the IPMI controller and host OS can access it), but I wouldn't recommend doing this.

Re: Proposed server purchase for GitLab.com

#214

Post by walrus01 »

gr2020 wrote: post #71

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the…

Something you should do to understand the actual power consumption of a server (ac power, in watts , at its input):

Build one node with the hardware configuration you intend to use. Same CPU, ram, storage.

Put it on a watt meter accurate to 1W.

Install Debian amd64 on it in a basic config and run 256 threads of the 'cpuburn' package while simultaneously running iozone disk bench and memory benchmarks.

This will give the figure of the absolute maximum load of the server, in watts, when it is running at 100% load on all cores, memory IO, and disk IO.

Watts = heat, since all electricity consumed in a data center is either being used to do physical work like spinning a fan, or is going to end up in the air as heat. Laws of physics. As whatever data center you're using will be responsible for cooling, this is not exactly your problem, but you should be aware of it if you're going to try to do something like 12 kilowatts density per rack.

Then multiply wattage of your prototype unit by number of servers. This will tell you how many fully loaded systems will fit in one 208v 30a circuit on the AC side.

Also, use the system bios option for power recovery to stagger bootup times by 10 seconds each per server, so that in the event of a total power loss and an entire rack of servers does not attempt to power up simultaneously.

Re: Proposed server purchase for GitLab.com

#215

Post by ihsw »

sytse wrote: post #185
sulam wrote: post #168

Earlier quoted context omitted.

We are also a SL customer -- 4-figures of hosts with them. We have had networking problems in the past (latency and loss far higher than I would expect to see in a well-provisioned DC) and talked to them about it. It ended up being contention with another customer, it got fixed, and our network performance has been great since. I would encourage you to look at what you can get without trying to do your own colo. You'…

Thanks for the suggestion. Any idea if they can offer 40 Gbps networking?

Few will offer 40Gbps without charging a pretty penny, generally the jump will go from 10Gbps to 100Gbps but not until it becomes cost-effective (and not anytime soon).

That said, SoftLayer does provide 20Gbps access within a private rack and 20Gbps access to the public network.

Re: Proposed server purchase for GitLab.com

#216

Post by virtuallynathan »

peterwwillis wrote: post #104

The first consideration should always be the DC, and the very last one is the software, after hardware, network, power and cooling. Where will my DC be? What kind of DC is it? What services do they provide? How long do I want it to take for an employee to get there, whether or not they have 24/7 remote hands? What kind of power resiliency do they provide? What will power cost? What kind of power do they provide per c…

>Network: i'm not a neteng, don't listen to me, but obviously it should be managed with nice fat switching fabric bandwidth, good forwarding rate and big uplink module support. 48-port switches don't always have the same bandwidth ratios as 24-port switches, and uplinks are much easier to manage on a 24-port than a 48.

With current gen ASICs and switches, this isn't generally true anymore. A $1000 48 port 1Gbps switch is fully non-blocking with almost 1:1 10G uplinks (48x1G in, 4x10G out)

Re: Proposed server purchase for GitLab.com

#217

Post by btgeekboy »

walrus01 wrote: post #214
gr2020 wrote: post #71

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the…

Something you should do to understand the actual power consumption of a server (ac power, in watts , at its input): Build one node with the hardware configuration you intend to use. Same CPU, ram, storage. Put it on a watt meter accurate to 1W. Install Debian amd64 on it in a basic config and run 256 threads of the 'cpuburn' package while simultaneously running iozone disk bench and memory benchmarks. This will give…

This is good advice. I'd consider not having them auto power on, however. This allows them to bring things up in a controlled manner.

That brings me to my next point: GitLab should also being mindful on what services are stored on which hardware - performing heroics to work around circular dependencies is the last thing you want to be doing when recovering from a power outage.

Re: Proposed server purchase for GitLab.com

#218

Post by Spooky23 »

kijiki wrote: post #184
Spooky23 wrote: post #114

I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…

If you're going to spend top dollar on Arista/Cisco/EMC/NetApp, you might as well stay in the cloud. None of the clouds use any of that super-expensive gear, so if you're going for cost savings, you'll need to use the same sort of commodity gear they use. Gitlab is obviously Linux-savvy and comfortable writing automation, so things like Cumulus Linux and minimal-handholding hardware vendors shouldn't cause them any i…

But this is a couple of hundred boxes, not AWS. I've been to a Microsoft data center... the scale is infinitely larger and solutions are different well.

My point isn't to knock them down. It takes cohones to be public about stuff like this. My instinct as a grumpy engineering director type is that there are holes here that need to be filled in.

Putting a major product at risk to save $30k against an Arista switch isn't a decision to make lightly. That means pricing the labor, upside benefit and business risk. If they are going to 100x this environment, Cumulus will save millions. If it will 3x, it will save a few thousand bucks -- who cares.

Re: Proposed server purchase for GitLab.com

#219

Post by walrus01 »

btgeekboy wrote: post #217
walrus01 wrote: post #214

Earlier quoted context omitted.

Something you should do to understand the actual power consumption of a server (ac power, in watts , at its input): Build one node with the hardware configuration you intend to use. Same CPU, ram, storage. Put it on a watt meter accurate to 1W. Install Debian amd64 on it in a basic config and run 256 threads of the 'cpuburn' package while simultaneously running iozone disk bench and memory benchmarks. This will give…

This is good advice. I'd consider not having them auto power on, however. This allows them to bring things up in a controlled manner. That brings me to my next point: GitLab should also being mindful on what services are stored on which hardware - performing heroics to work around circular dependencies is the last thing you want to be doing when recovering from a power outage.

Yes, this is a choice that depends on the facility, and what sort of recovery plan you have for total power failure. In many cases you would want to have everything remain off, and have a remote hands person power up certain Network equipment, and key service, before everything else. On the other hand, you may want to design everything to recover itself, without any button pushing from humans.

Depends a lot of server/HA/software architecture.

Re: Proposed server purchase for GitLab.com

#220

Post by walrus01 »

MichaelGG wrote: post #137
wmf wrote: post #83

I have built out a few racks of Supermicro twins. In general I would suggest hiring your ops people first and then letting them buy what they are comfortable with. C2: The Dell equivalent is C6320. CPU: Calculate the price/performance of the server , not the processor alone. This may lead you towards fewer nodes with 14-core or 18-core CPUs. Disk: I would use 2.5" PMR (there is a different chassis that gives 6x2.5" p…

Can you elaborate on what deficiencies 10GBase-T has in server applications?

One, category 6A cables actually cost two and a half times as much as a basic single mode fiber patch cables. Two, cable diameter. Ordinary LC to LC fiber cables with 2 millimeter diameter duplex fiber are much easier to manage then category 6A. Three, choice of network equipment. There is a great deal more equipment that will take ordinary sfp+ 10 gig transceivers, then equipment that has 10 gigabit copper ports. As a medium-sized ISP, I rarely if ever see Copper 10 gigabit.
Post reply on HN