Proposed server purchase for GitLab.com • Hacker News Board

Re: Proposed server purchase for GitLab.com

#251

Post by daxelrod » Mon, Dec 12, 2016, 3:37 AM UTC

closeparen wrote: post #249

daxelrod wrote: post #211

Earlier quoted context omitted.

> they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that they can expose over NFS As someone who administers GitLab for my company, yes please. Any high availability scenario that involves "just mount the same NFS volume on your standby" is a nonstarter for us. (We've found mounting NFS across di…

>Any high availability scenario that involves "just mount the same NFS volume on your standby" is a nonstarter for us What are the alternatives? I suppose there's MySQL's "stream asynchronously to the standby at the application level." ... which, now that I think about it, should be pretty easy to do with Git, since pushing and pulling discrete units of data are core concepts...

Yep! See GitHub's Spokes, née DGit. http://githubengineering.com/building-resilience-in-spokes/

Re: Proposed server purchase for GitLab.com

#252

Post by _delirium » Mon, Dec 12, 2016, 3:41 AM UTC

JustARandomGuy wrote: post #204

Spooky23 wrote: post #114

I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…

Thank goodness someone said it. Perhaps I don't understand the problem domain, but I don't understand why CephFS is being considered for this task. You're trying to treat your entire set of files across all repos as a single filesystem, but that's an entirely incorrect assumption. The I/O activity on one repo/user does not affect the I/O activity of an entirely different user. Skip the one filesystem idea, shard base…

Treating the whole thing as one FS is the current architecture GitLab uses, so is more of an existing constraint than a proposed architecture. To get distributed storage you either need to rewrite GitLab to deal with distributed storage, or run it on another layer that presents an illusion of one big FS (whether that's CephFS or a storage appliance).

Re: Proposed server purchase for GitLab.com

#253

Post by joneholland » Mon, Dec 12, 2016, 3:50 AM UTC

Disregarding the obvious signs that there is severe lack of experience from gitlab staff on the physical DC build, they completely underestimate how difficult it is to run ceph at scale with a reliable SLA.

Re: Proposed server purchase for GitLab.com

#254

Post by hhw » Mon, Dec 12, 2016, 3:55 AM UTC

walrus01 wrote: post #220

MichaelGG wrote: post #137

Earlier quoted context omitted.

Can you elaborate on what deficiencies 10GBase-T has in server applications?

One, category 6A cables actually cost two and a half times as much as a basic single mode fiber patch cables. Two, cable diameter. Ordinary LC to LC fiber cables with 2 millimeter diameter duplex fiber are much easier to manage then category 6A. Three, choice of network equipment. There is a great deal more equipment that will take ordinary sfp+ 10 gig transceivers, then equipment that has 10 gigabit copper ports. As…

SFP+ 10Gbase-T 3rd party optics started hitting the market this year, but none of the major switch vendors offer them yet, so the coding effectively lies about what cable type it is. Just an option to keep in your back pocket, as onboard 10Gb is typically 10Gbase-T. Thankfully, onboard SFP+ is becoming more common however.

For short distances of known length, twinax cables which are technically copper can be used. They're thinner than regular cat6a but only about the same as thin 6a and thicker than typical unshielded Duplex fiber patch cables. Twinax can be handy if connecting Arista switches to anything else that restricts 3rd party optics, as Arista only restricts other cable types. Twinax is also the cheapest option.

Re: Proposed server purchase for GitLab.com

#255

Post by favadi » Mon, Dec 12, 2016, 3:56 AM UTC

sytse wrote: post #103

dsr_ wrote: post #61

Z1: the word "monitoring" does not appear in this document. You will need to monitor: - ping - latency - temperatures - cpu utilization - ram utilization - disk utilization - disk health - context switches - IP addresses assigned and reaching expected MACs - appropriate ports open and listening - appropriate responses - time to execute queries - processes running - process health - at least something for every bit of…

Good point, we know monitoring is very important. Our Infrastructure lead Pablo will do a webinar of our Prometheus monitoring soon https://page.gitlab.com/20161207_PrometheusWebcast_LandingPa... We're bundling Prometheus with GitLab https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1481 Brian Brazil is helping us https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1481#not... On January 9 our Prometheus lead will…

Gitlab package includes too many things! I remember Gitlab sponsors a guy to do proper packaging to include Gitlab in debian last year, not sure what is the status.

Re: Proposed server purchase for GitLab.com

#256

Post by closeparen » Mon, Dec 12, 2016, 4:07 AM UTC

samstave wrote: post #147

Spooky23 wrote: post #134

Earlier quoted context omitted.

Why would you care?

What's your email address, SSN and password. If you have nothing to hide, then you shouldn't care why I want to read all your messages.

Why would you care, as in, why would you have cleartext data on the wire?

This is Gitlab. There should never be any data in or out except HTTPS and SSH.

Re: Proposed server purchase for GitLab.com

#257

Post by late2part » Mon, Dec 12, 2016, 4:11 AM UTC

DanielDent wrote: post #240

late2part wrote: post #210

Earlier quoted context omitted.

note that you can get around the 80% breaker limit by having your DC hardwire the power, if you have enough scale to have them do this for you.

Circuit breakers aren't just there for the amusement value of watching a clustered system go into split-brain mode... I.e. you would also want to be sure that your wiring was rated for 100% utilization, and that other circuit-breaker-like functions exist. Fire is an actual thing, and figuring out the best way to recharge a halon system isn't exactly what you want to be doing.

Yes. That's why they hardwire and use a breaker rated at 100%. In most jurisdiction, code requires breaker at 80% if a plug/receptacle is used. If you are hardwired you can use 100% of your capacity before tripping the breaker. You have incorrectly assumed that I suggested that you ignore breakers.

When you hardwire the circuit the electrical code allows you to use a 100% breaker.

Re: Proposed server purchase for GitLab.com

#258

Post by late2part » Mon, Dec 12, 2016, 4:14 AM UTC

mohctp wrote: post #243

late2part wrote: post #210

Earlier quoted context omitted.

note that you can get around the 80% breaker limit by having your DC hardwire the power, if you have enough scale to have them do this for you.

This really is only a concern when you are paying a monthly recurring charge (MRC) by the breaker amp with many power drops. For a deployment of this scale it should be metered power (For example 1 (or more) 3phase a+b drops to each cabinet) where you only pay a Non-Recurring setup Charge (NRC) and then the MRC is based on actual power draw. 3phase also means fewer physical PDU's (uses less space), but more physical…

Not really, although I agree with your reasoning. The other is issue is capex. When I deploy 240kW pods, if I use 80% breakers, I have to deploy 25% more PDUs than if I have 100% breakers.

Since my cabinet number is usually evenly divisible by N*PDUs, this impacts overall capital.

Re: Proposed server purchase for GitLab.com

#259

Post by late2part » Mon, Dec 12, 2016, 4:15 AM UTC

closeparen wrote: post #250

late2part wrote: post #206

Earlier quoted context omitted.

I disagree. With over 300 servers in AWS, you can almost certainly build a redundant data center with hardware at less than 60% of the costing assuming 3 year depreciation. Arista and Cisco shouldn't cost top dollar; though anyone buying EMC or Netapp for any new build should have their union card revoked. FreeNAS ftw uber alles. Source: Did it twice.

Is FreeNAS something people actually run Serious Business, at-scale production datacenters on? I've run it in my home a few times out of curiosity, and that was never my impression.

Yes. I know of several billion dollar companies that run it in a web facing production operations capacity.

Re: Proposed server purchase for GitLab.com

#260

Post by late2part » Mon, Dec 12, 2016, 4:20 AM UTC

etcet wrote: post #248

late2part wrote: post #202

I recently built some data center processing using SuperMicro Twin-Nodes^2 [0] and MicroBlades [1]. We're setup for 38kw [a] possible gross wattage using the dual-node blades using the Xeon D-1541 [2]. The amazing thing about the D-1541 blades is we get around 100W per server, with 8 hyperthreaded cores, and a 3.84T SSD. With the 6U chassis, you have 28 blades with 2 nodes each - 56 medium sized servers for 5.6kw in…

If not Ceph, what are you using for storage?

2RU SuperMicro server with 24xSSD with FreeNAS raid10 or raidz2 exporting volumes via iScsi.

Separately I'm using a FreeNAS controller with 4 SAS HBAs supporting 3 JBODs with 45 8TB HGST He8 near line SATA disks each (135 disks or ~1PB) for backups and slow data.