Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

151–160 of 327 posts

Re: Proposed server purchase for GitLab.com

#151

Post by Spooky23 »

manacit wrote: post #144
Spooky23 wrote: post #114

I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…

I'm happy to see this - I could not agree more with these points. I think they are coming at this problem from the wrong perspective - instead of growing from virtual servers to their own dedicated hardware to get better CephFS performance, they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that t…

> As someone who has run large scale Ceph before (though not CephFS, thankfully), it's not easy to run at scale. We had a team of 5 software engineers as well as an Ops and Hardware team, and we had to do a lot to get it stable. It's not as easy as installing it and walking away.

This is a really good point. That's easily $1M in payroll. You could probably run a decent tiered SAN with 80-95% fewer labor dollars. Plus have the benefit of torturing the vendor when you hit scaling hiccups.

Re: Proposed server purchase for GitLab.com

#152

Post by __d »

sytse wrote: post #84
spydum wrote: post #78

Earlier quoted context omitted.

fully agreed here, especially leave yourself room for when you misjudged some resource utilization (and need more). Nothing worse than having a resource crunch (cpu/mem/io) and not being able to resolve it because your rack is out of power/cooling/etc - you'll come to appreciate how easy it was in cloud just clicking the button and turning out your wallet.

Great points. We'll make sure to wire to separate power feeds that can both handle the entire load. Suggestions in how to calculate this? Taking the maximum rated load seems over the top.

For HP gear, use HP Power Advisor utility. For Dell, see the Data Center Capacity Planner. Not sure what SuperMicro has -- check with your VAR.

Re: Proposed server purchase for GitLab.com

#153

Post by sytse »

gr2020 wrote: post #124
gr2020 wrote: post #71

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the…

I just remembered a blog post I wrote a while back about exactly these points: http://www.rassoc.com/gregr/weblog/2011/04/13/choosing-a-col...

Thanks!

Re: Proposed server purchase for GitLab.com

#154

Post by sytse »

pyrox420 wrote: post #100
sytse wrote: post #97

Earlier quoted context omitted.

We looked at providers such as Softlayer but while they guarantee the performance of the servers they typically can't guaranty network latency. Since we're doing this to reduce latency https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ this is essential to us. We'll be glad to look into alternatives that manage the servers and network for us although the argument in https://news.ycombinator.com/item?id=131534…

Been a softlayer customer for 4 years now. Their network is pretty awesome. When hosted in the same datacenter it's sub-ms response time always. If there is an issue they get right on it. You can even ask them to host the stuff in the same rack to get even better response time.

Cool, I was not aware you can ask for same rack hosting.

Re: Proposed server purchase for GitLab.com

#155

Post by davidbrent »

gr2020 wrote: post #71

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the…

Think much harder about power and cooling. A few points:

I've only ever built desktop machines, and this top comment drew a surprising parallel to most help me with my desktop build type posts. Granted, I'm sure as you dig deeper, the reasoning may be much different, but myself being ignorant about a proper server build, it was somehow reassuring to see power and cooling at the top!

Re: Proposed server purchase for GitLab.com

#156

Post by AnonNo15 »

I would say you should start with hiring 2 full-time senior ops and 2 senior network admins.

Then give them freedom to do all hardware picks AND hiring some more intermediate/junior ops/admin staff.

You can do with 3-4 ops and maybe 3 network admins people in total.

Re: Proposed server purchase for GitLab.com

#157

Post by dgemm »

This undertaking seems like a huge investment in time & overhead for only 64 servers. My guess is that with this move, performance will go up and availability will go down.

I can understand the need for performance but if it were my business I would have taken a significantly different approach.

Re: Proposed server purchase for GitLab.com

#158

Post by bronson »

lykron wrote: post #107
wmf wrote: post #102

Earlier quoted context omitted.

This is one rack of equipment; no spines are needed. http://blog.ipspace.net/2014/10/all-you-need-are-two-top-of-... Cumulus isn't really SDN so I'm not sure what you're saying there. Traditional networking is fine but it's totally different than Linux so you need dedicated netops people to manage it. And Cisco is the most expensive traditional vendor.

I'm not super well-versed in the networking area. I misread the posting. I thought they were looking at 64Us of servers. I've just gotten the notion that SDN is almost ready for primetime, just not yet.

It's been almost ready for primetime for over a decade now.

Re: Proposed server purchase for GitLab.com

#159

Post by Spooky23 »

samstave wrote: post #147
Spooky23 wrote: post #134

Earlier quoted context omitted.

Why would you care?

What's your email address, SSN and password. If you have nothing to hide, then you shouldn't care why I want to read all your messages.

Where are the customers coming from that are immune to national intelligience apparatus?

I wouldn't make a business decision around a general feeling of paranoia. German spies aren't an improvement over American or British ones.

Re: Proposed server purchase for GitLab.com

#160

Post by sytse »

viraptor wrote: post #138

As someone already mentioned "all the other services" are missing (dns, ntp, monitoring, etc.), but also: - Shouldn't there be a puppet / chef / whatever deployment coordinator in there somewhere? - There's no mention of a virtualisation environment. While it's not a hardware issue really, all the extra services mentioned before will not take the whole server and you'll want to collocate some of them. (maybe even som…

- We would continue to use our existing cloud hosted Chef server for now.

- We want to use Kubernetes instead of virtualization.

- We provisioned 2 spare database servers for that reasons.

- Git already compresses files with zlib so I'm not sure we can compress it much further https://git-scm.com/book/uz/v2/Git-Internals-Packfiles

- The servers have a separate management port and "Apart from those routers we'll have a separate router for a 1Gbps management network."

- I agree we'll probably need more spare servers.

Post reply on HN