Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

201–210 of 327 posts

Re: Proposed server purchase for GitLab.com

#201

Post by msandford »

halbritt wrote: post #199
msandford wrote: post #192

Earlier quoted context omitted.

Also whatever you do, don't buy all one kind of disk. That'll be the thing that dies first and most frequently. Buy from different manufacturers and through different vendors to try and get disks from at least a few different batches. That way you don't get hit by some batch of parts being out of spec by 5% instead of 2% and them all failing within a year, all at the same time. If you do somehow manage to pick the pe…

I had this problem with the Supermicro SATA DOMs. Had problems with the whole batch. That said, I bought the same 6TB HGST disk for two years.

As long as you're not buying all the disks at once sticking with one manufacturer and brand should be fine. If you're buying 25% of your total inventory every year it'll all be spread out to just a few percent per month.

But when you're buying 100% of your disk inventory at once there's a serious "all eggs in one basket" risk.

Re: Proposed server purchase for GitLab.com

#202

Post by late2part »

I recently built some data center processing using SuperMicro Twin-Nodes^2 [0] and MicroBlades [1].

We're setup for 38kw [a] possible gross wattage using the dual-node blades using the Xeon D-1541 [2]. The amazing thing about the D-1541 blades is we get around 100W per server, with 8 hyperthreaded cores, and a 3.84T SSD. With the 6U chassis, you have 28 blades with 2 nodes each - 56 medium sized servers for 5.6kw in 6U - under a kw per RU.

For your 70 some server workloads, I'd recommend using the microblades.

For your higher lever workloads, I think the SuperMicro TwinNode makes sense.

Be very very careful about Ceph hype. Ceph is good at redundancy and throughput, but not at iops, and rados iops are poor. We couldn't get over 60k randrw iops across a 120 OSD cluster with 120 SSDs.

For your NFS server, I'd recommend a large FreeNAS system, put a big brain in it and throw spinning platters.

Datacenters can/will do your 30kw

[0] - https://www.supermicro.com/products/nfo/2UTwin2.cfm [1] - https://www.supermicro.com/products/MicroBlade/ [2] - https://www.supermicro.com/products/MicroBlade/module/MBI-62... [a] - Although we have 38kw of possible power there, it's practically well under the 27kw we'll get with 4x208v@60A PDUs at 80%.

Re: Proposed server purchase for GitLab.com

#203

Post by late2part »

gr2020 wrote: post #71

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the…

Nowadays once you get past 10 racks or 50kw, you generally only pay for power - the space/cooling/etc is "free" as your limiting rate is power and the vendor's ability to move thermal. You'll likely want a chimney cabinet like the Chatsworth GlobalFrame [1].

[1] - http://www.chatsworth.com/products/cabinet-and-enclosure-sys...

Re: Proposed server purchase for GitLab.com

#204

Post by JustARandomGuy »

Spooky23 wrote: post #114

I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…

Thank goodness someone said it.

Perhaps I don't understand the problem domain, but I don't understand why CephFS is being considered for this task. You're trying to treat your entire set of files across all repos as a single filesystem, but that's an entirely incorrect assumption. The I/O activity on one repo/user does not affect the I/O activity of an entirely different user. Skip the one filesystem idea, shard based on user/location/whatever.

I'd appreciate any comments explaining why I'm wrong, because this doesn't seem to be a productive design to me.

Re: Proposed server purchase for GitLab.com

#205

Post by late2part »

wmf wrote: post #83

I have built out a few racks of Supermicro twins. In general I would suggest hiring your ops people first and then letting them buy what they are comfortable with. C2: The Dell equivalent is C6320. CPU: Calculate the price/performance of the server , not the processor alone. This may lead you towards fewer nodes with 14-core or 18-core CPUs. Disk: I would use 2.5" PMR (there is a different chassis that gives 6x2.5" p…

I politely disagree on the 10GBaseT. I built out hundreds of nodes with TOR 10G using Arista switches and it worked great.

Re: Proposed server purchase for GitLab.com

#206

Post by late2part »

kijiki wrote: post #184
Spooky23 wrote: post #114

I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…

If you're going to spend top dollar on Arista/Cisco/EMC/NetApp, you might as well stay in the cloud. None of the clouds use any of that super-expensive gear, so if you're going for cost savings, you'll need to use the same sort of commodity gear they use. Gitlab is obviously Linux-savvy and comfortable writing automation, so things like Cumulus Linux and minimal-handholding hardware vendors shouldn't cause them any i…

I disagree. With over 300 servers in AWS, you can almost certainly build a redundant data center with hardware at less than 60% of the costing assuming 3 year depreciation.

Arista and Cisco shouldn't cost top dollar; though anyone buying EMC or Netapp for any new build should have their union card revoked. FreeNAS ftw uber alles.

Source: Did it twice.

Re: Proposed server purchase for GitLab.com

#207

Post by zonywhoop »

sytse wrote: post #84
spydum wrote: post #78

Earlier quoted context omitted.

fully agreed here, especially leave yourself room for when you misjudged some resource utilization (and need more). Nothing worse than having a resource crunch (cpu/mem/io) and not being able to resolve it because your rack is out of power/cooling/etc - you'll come to appreciate how easy it was in cloud just clicking the button and turning out your wallet.

Great points. We'll make sure to wire to separate power feeds that can both handle the entire load. Suggestions in how to calculate this? Taking the maximum rated load seems over the top.

Having designed and run colo data centers for many years my rule of thumb for calculating this for customers was to use the vendors tools if available or take 80% of power supply rating. Keep in mind that most servers will have a peak load during initial power on as the fans and components come online and run through testing. If a rack ever comes up on a single channel and the circuits are not rated right that breaker will just pop right back offline and you'll have to bring it up by unplugging servers and bringing them up in sets. Also most breakers are only rated at 80% load so you have to de-rate them for your load. So, e.g., a 20amp x 208 3 phase circuit really only has 8KW of constant power draw available.

Re: Proposed server purchase for GitLab.com

#208

Post by late2part »

sytse wrote: post #84
spydum wrote: post #78

Earlier quoted context omitted.

fully agreed here, especially leave yourself room for when you misjudged some resource utilization (and need more). Nothing worse than having a resource crunch (cpu/mem/io) and not being able to resolve it because your rack is out of power/cooling/etc - you'll come to appreciate how easy it was in cloud just clicking the button and turning out your wallet.

Great points. We'll make sure to wire to separate power feeds that can both handle the entire load. Suggestions in how to calculate this? Taking the maximum rated load seems over the top.

you want 2*N Power feeds where N = 1 or 2. I recommend ServerTech PDUs. You'll want to factor in the maximum load, but with the amount of servers and power you're doing, you should be able to negotiate metered power at under $250/kW/month. Give them a commit that's 1/3 of your TDW/reportd power load, and then pay for what you use.

Re: Proposed server purchase for GitLab.com

#209

Post by late2part »

zonywhoop wrote: post #207
sytse wrote: post #84

Earlier quoted context omitted.

Great points. We'll make sure to wire to separate power feeds that can both handle the entire load. Suggestions in how to calculate this? Taking the maximum rated load seems over the top.

Having designed and run colo data centers for many years my rule of thumb for calculating this for customers was to use the vendors tools if available or take 80% of power supply rating. Keep in mind that most servers will have a peak load during initial power on as the fans and components come online and run through testing. If a rack ever comes up on a single channel and the circuits are not rated right that breake…

+1. As tempting as it is to say "oh this server only needs full power when it boots" this may come back to bite you. As you grow usage in CPU and Disk, the power used by the server will increase substantially.

Re: Proposed server purchase for GitLab.com

#210

Post by late2part »

zonywhoop wrote: post #207
sytse wrote: post #84

Earlier quoted context omitted.

Great points. We'll make sure to wire to separate power feeds that can both handle the entire load. Suggestions in how to calculate this? Taking the maximum rated load seems over the top.

Having designed and run colo data centers for many years my rule of thumb for calculating this for customers was to use the vendors tools if available or take 80% of power supply rating. Keep in mind that most servers will have a peak load during initial power on as the fans and components come online and run through testing. If a rack ever comes up on a single channel and the circuits are not rated right that breake…

note that you can get around the 80% breaker limit by having your DC hardwire the power, if you have enough scale to have them do this for you.
Post reply on HN