Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

241–250 of 327 posts

Re: Proposed server purchase for GitLab.com

#241
There are two major pitfalls to crowd-sourced consulting such as this.

1) Contributors have not been vetted - Some responses are based on real world experience, and some is conjecture from arm-chair quarterbacks. (A simple example would be that nobody has mentioned with any of the SuperMicro 2U Twins that you have to be cautious about the PDU models and outlet locations of 0U PDU's to not block node service/replacement in the rack)

2) There are multiple ways to skin a cat - There are many viable solutions in this thread, but you can't simply take a little bit of this, a little bit of that, and piece together a new platform that "just works." -- Better to go with a know working solution than a little of this and a little of that. Multiple drivers tend to be less effective.

I am the owner of a bespoke Infrastructure as a Service provider that delivers solutions to sites of similar metrics, and speak with plenty of real-world experience.

Larger Providers - We find a lot of clients move away from AWS, Softlayer, Rackspace, et al. as the larger providers aren't nearly as interested in working with the less-than-standard configurations. They want highly-repeatable business, not one-off solutions.

I'd love to talk in more depth with you about how we can deliver exactly what you need based on years of experience in delivering highly customized solutions. We'll save you money and headache.

Re: Proposed server purchase for GitLab.com

#242

Earlier quoted context omitted.

Why does latency matter for fin tech for a repository / source control?

"more things than just latency" Some financial institutions require all sensitive* data to be stored/hosted in the same country/state(archaic yes). It's real hard to actually define sensitive data but the IP * in some code a quant wrote can totally be considered a trade secret by a non technical person Please don't get me started on how stupid I think it is that people consider code to be IP

>archaic yes

Definitely not archaic. If the FISA courts taught you anything, it should be that the country your things are living in determines which entities can tap your traffic/hardware.

Re: Proposed server purchase for GitLab.com

#243

Earlier quoted context omitted.

Having designed and run colo data centers for many years my rule of thumb for calculating this for customers was to use the vendors tools if available or take 80% of power supply rating. Keep in mind that most servers will have a peak load during initial power on as the fans and components come online and run through testing. If a rack ever comes up on a single channel and the circuits are not rated right that breake…

note that you can get around the 80% breaker limit by having your DC hardwire the power, if you have enough scale to have them do this for you.

This really is only a concern when you are paying a monthly recurring charge (MRC) by the breaker amp with many power drops.

For a deployment of this scale it should be metered power (For example 1 (or more) 3phase a+b drops to each cabinet) where you only pay a Non-Recurring setup Charge (NRC) and then the MRC is based on actual power draw.

3phase also means fewer physical PDU's (uses less space), but more physical breakers. Over-building delivery capability will eliminate any over-draw concerns for startup cycles.

Re: Proposed server purchase for GitLab.com

#244
post #94
post #83

I have built out a few racks of Supermicro twins. In general I would suggest hiring your ops people first and then letting them buy what they are comfortable with. C2: The Dell equivalent is C6320. CPU: Calculate the price/performance of the server , not the processor alone. This may lead you towards fewer nodes with 14-core or 18-core CPUs. Disk: I would use 2.5" PMR (there is a different chassis that gives 6x2.5" p…

With regards to the switches, I would argue that they should skip SDN switches all together and get some Cisco Catalysts as TOR Switches. 2x 48-port switches for each rack with redundant core routers in spine-leaf. SDN is cool, but at the current scale, it seems like it would be more resource intensive than it is worth.

Agreed, skip SDN. Skip MC-LAG.

Juniper, Cisco, and Arista all have solutions for this environment.

Re: Proposed server purchase for GitLab.com

#245

> We want to dual bound the network connections to increase performance and reliability. This will allow us to take routers out of service during low traffic times, for example to restart them after a software upgrade. does not really agree with > Each of the two physical network connections will connect to a different top of rack router. Sure, you can do it with something like MLAG, but that's really just moving you…

>> N1 Which router should we purchase? >Pick your favorite. For what you're looking for here, everything is largely using the same silicon (broadcom chipsets).

For switches, yes. Many of the switches share the same merchant silicon Broadcom Trident-II, Tomahawk, et al., however there are switches like the Juniper EX9200 which isn't baed on merchant silicon. Routers (N1) are also not typically based on merchant silicon (Juniper Trio-3D for example).

Re: Proposed server purchase for GitLab.com

#246

The first consideration should always be the DC, and the very last one is the software, after hardware, network, power and cooling. Where will my DC be? What kind of DC is it? What services do they provide? How long do I want it to take for an employee to get there, whether or not they have 24/7 remote hands? What kind of power resiliency do they provide? What will power cost? What kind of power do they provide per c…

>Network: i'm not a neteng, don't listen to me, but obviously it should be managed with nice fat switching fabric bandwidth, good forwarding rate and big uplink module support. 48-port switches don't always have the same bandwidth ratios as 24-port switches, and uplinks are much easier to manage on a 24-port than a 48. With current gen ASICs and switches, this isn't generally true anymore. A $1000 48 port 1Gbps switc…

agreed. There is absolutely zero reason to even consider 24 port switches in this environment.

Re: Proposed server purchase for GitLab.com

#247
post #197
post #160

Earlier quoted context omitted.

- We would continue to use our existing cloud hosted Chef server for now. - We want to use Kubernetes instead of virtualization. - We provisioned 2 spare database servers for that reasons. - Git already compresses files with zlib so I'm not sure we can compress it much further https://git-scm.com/book/uz/v2/Git-Internals-Packfiles - The servers have a separate management port and "Apart from those routers we'll have…

I hope to god you are not using Kubernetes for your databases.

Why not? Google Cloud SQL runs on Kubernetes.

Re: Proposed server purchase for GitLab.com

#248

I recently built some data center processing using SuperMicro Twin-Nodes^2 [0] and MicroBlades [1]. We're setup for 38kw [a] possible gross wattage using the dual-node blades using the Xeon D-1541 [2]. The amazing thing about the D-1541 blades is we get around 100W per server, with 8 hyperthreaded cores, and a 3.84T SSD. With the 6U chassis, you have 28 blades with 2 nodes each - 56 medium sized servers for 5.6kw in…

If not Ceph, what are you using for storage?

Re: Proposed server purchase for GitLab.com

#249

Earlier quoted context omitted.

I'm happy to see this - I could not agree more with these points. I think they are coming at this problem from the wrong perspective - instead of growing from virtual servers to their own dedicated hardware to get better CephFS performance, they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that t…

> they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that they can expose over NFS As someone who administers GitLab for my company, yes please. Any high availability scenario that involves "just mount the same NFS volume on your standby" is a nonstarter for us. (We've found mounting NFS across di…

>Any high availability scenario that involves "just mount the same NFS volume on your standby" is a nonstarter for us

What are the alternatives?

I suppose there's MySQL's "stream asynchronously to the standby at the application level."

... which, now that I think about it, should be pretty easy to do with Git, since pushing and pulling discrete units of data are core concepts...

Re: Proposed server purchase for GitLab.com

#250
post #184

Earlier quoted context omitted.

If you're going to spend top dollar on Arista/Cisco/EMC/NetApp, you might as well stay in the cloud. None of the clouds use any of that super-expensive gear, so if you're going for cost savings, you'll need to use the same sort of commodity gear they use. Gitlab is obviously Linux-savvy and comfortable writing automation, so things like Cumulus Linux and minimal-handholding hardware vendors shouldn't cause them any i…

I disagree. With over 300 servers in AWS, you can almost certainly build a redundant data center with hardware at less than 60% of the costing assuming 3 year depreciation. Arista and Cisco shouldn't cost top dollar; though anyone buying EMC or Netapp for any new build should have their union card revoked. FreeNAS ftw uber alles. Source: Did it twice.

Is FreeNAS something people actually run Serious Business, at-scale production datacenters on?

I've run it in my home a few times out of curiosity, and that was never my impression.

Post reply on HN