Proposed server purchase for GitLab.com • Hacker News Board

Re: Proposed server purchase for GitLab.com

#221

Post by AstroJetson » Mon, Dec 12, 2016, 1:27 AM UTC

daenney wrote: post #21

AstroJetson wrote: post #3

Love how they did some number crunching and decided that rent vs own, own won. I think that if more places looked they would find that out also. There must be a margin in it since the big players are making money at it. I'm interested to see what they end up with in the end.

> Love how they did some number crunching and decided that rent vs own, own won. I think that if more places looked they would find that out also I wouldn't be so quick to jump to that conclusion. It's not just the cost of owning and renewing the hardware, it's everything else that comes with it. Designing your network, performance tuning and debugging everything. Suddenly you have a capacity issue, now what b/c you'…

So in gitlabs case, they have a load that they can monitor and predict. They are looking at 60+ processors, so they can plan to add 10% (6 procs at a time) and grow. They know their load, so the likely need to spin 150% of current capacity isn't something on their plan. I'll give you that there are companies that have erratic loads that are hard to predict, they make sense to place in something that can grow 100% on an email.

Big companies, most of their servers have a pretty stable load, it's unlikely things like internal email, Sharepoint, ERP/MAP systems will take a spike. It's only things like front end order processing that takes the hit.

There are lots of businesses that make sense and some that don't

I like the concept of "racing to the bottom" but they are still making money. But lets take your comment of the Value Added Services other than the ability to spin up capacity. What's the cost to Gitlab to pull this together and keep it running? There is an overflow every day on HN articles about operations monitors, containers, network monitoring. The tools are there, its an effort to glue them together, but then they are there.

So I'll still posit there is cases that the dollars to own are less than the dollars to rent. And I'll agree with your cases of rent because of capacity blowouts is key. The issue, is your ops team savvy enough to figure out what to keep/own, what to rent?

Re: Proposed server purchase for GitLab.com

#222

Post by walrus01 » Mon, Dec 12, 2016, 1:27 AM UTC

late2part wrote: post #203

gr2020 wrote: post #71

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the…

Nowadays once you get past 10 racks or 50kw, you generally only pay for power - the space/cooling/etc is "free" as your limiting rate is power and the vendor's ability to move thermal. You'll likely want a chimney cabinet like the Chatsworth GlobalFrame [1]. [1] - http://www.chatsworth.com/products/cabinet-and-enclosure-sys...

This does depend a lot on geographical location. You're not going to get free racks in a major carrier exchange point in Manhattan, or downtown Seattle, or close to the urban core of San Francisco. There will definitely be a significant quantity discount, as you increase in numbers, and a lot of your cost will be power and not racks.

In somewhere that is very close (OSI layer 1) topologically to a major traffic interexchange point, you will definitely be paying somehow for the monthly cost of the square footage occupied. For example a colo in 60 Hudson or 25 Broadway in NYC, or in one Wilshire in LA.

Large-scale colocation pricing that is based on power only will be found in locations that are not also major traffic inter exchange points. For example quincy WA or the many data centers in suburban new jersey.

Re: Proposed server purchase for GitLab.com

#223

Post by predakanga » Mon, Dec 12, 2016, 1:28 AM UTC

samstave wrote: post #46

nikanj wrote: post #27

Earlier quoted context omitted.

I remain skeptical about the existence of non-tapped lines.

+1 EDIT: Funny, every single documented, proven link for the last several years has proven me and GP correct and all you nay-sayers wrong.

I believe the downvotes are due to "+1" not adding anything to the conversation, as opposed to people disagreeing with the sentiment.

Re: Proposed server purchase for GitLab.com

#224

Post by comex » Mon, Dec 12, 2016, 1:31 AM UTC

btgeekboy wrote: post #93

deadbunny wrote: post #7

Yeah, link aggregation doesn't work how they think it does. And not having a separate network for Ceph is going to bite them in the arse. GitLab is fine software but fuck me, they need to hire someone with actual ops experience (based on this post and their previous "we tried running a clustered file system in the cloud and for some reason it ran like shit" post).

They'd save themselves a whole lot of time, effort, and money if they looked at partitioning their data storage instead of plowing ahead with Ceph. They have customers with individual repositories. There is no need to have one massive filesystem / blast radius.

I don't know much about this stuff, but won't that stop working if they ever decide to expand to multiple geographical sites, to reduce latency to customers in different locations? In that case, different sites can receive requests for the same repositories, and ideally each site would be able to provide read only access without synchronization, with some smarts for maintaining caches, deciding which site should 'own' each file, etc. They could roll their own logic for that, but doesn't that pretty much exactly describe the job of a distributed filesystem? So they'd end up wanting Ceph anyway, so they may as well get experience with it now.

Re: Proposed server purchase for GitLab.com

#225

Post by kyledrake » Mon, Dec 12, 2016, 1:35 AM UTC

manacit wrote: post #144

Spooky23 wrote: post #114

I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…

I'm happy to see this - I could not agree more with these points. I think they are coming at this problem from the wrong perspective - instead of growing from virtual servers to their own dedicated hardware to get better CephFS performance, they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that t…

> As someone who has run large scale Ceph before (though not CephFS, thankfully), it's not easy to run at scale. We had a team of 5 software engineers as well as an Ops and Hardware team, and we had to do a lot to get it stable. It's not as easy as installing it and walking away.

Can you give some examples of the problems you ran into?

Re: Proposed server purchase for GitLab.com

#226

Post by kabdib » Mon, Dec 12, 2016, 1:36 AM UTC

wmf wrote: post #83

I have built out a few racks of Supermicro twins. In general I would suggest hiring your ops people first and then letting them buy what they are comfortable with. C2: The Dell equivalent is C6320. CPU: Calculate the price/performance of the server , not the processor alone. This may lead you towards fewer nodes with 14-core or 18-core CPUs. Disk: I would use 2.5" PMR (there is a different chassis that gives 6x2.5" p…

I too have built out several racks of SuperMicro twins.

I'm never buying another SuperMicro, for many reasons. The amount of cabling for properly redundant connections is a killer; it's at least three cables per system (five in our case); a rack will have hundreds of wires to manage. Access to the blade is in the back, where the cables are, and you have to think ahead and route things cleverly if you want be able to remove blades later.

The comment about doing something other than three 48-port switches is bang on. And if you're running Juniper hardware, avoid the temptation to make a Virtual Chassis (because it becomes a single point of failure, honest, and you will hate yourself when you have to do a firmware upgrade).

19KW is still a ton of power, and I'm surprised the datacenter isn't worried (none of the datacenters we use worldwide go much above 16KW usable). Also, you need to make sure you're on redundant circuits, and that things still work one of the power legs totally off. Make sure you know what kind of PDU you need (two-phase or three-phase), and that you load the phases equally.

Re: Proposed server purchase for GitLab.com

#227

Post by spullara » Mon, Dec 12, 2016, 1:37 AM UTC

All of your data and IOPs could be handled by a single Pure Storage FlashBlade. This would significantly simplify almost everything about your system. (I'm an investor, but generally wouldn't point this out unless someone is seriously considering on premise deployment.)

Re: Proposed server purchase for GitLab.com

#228

Post by daxelrod » Mon, Dec 12, 2016, 1:42 AM UTC

sytse wrote: post #131

theptip wrote: post #74

Earlier quoted context omitted.

That's a fair comment RE: outsourcing, but at my company I'd bias towards bringing some distributed computing knowledge in-house rather than bringing ops expertise plus maintenance burden in-house; sounds like you're going to have to add new expertise to your team either way. Worth investigating if you can bolt on a distributed datastore like etcd or ZooKeeper to store the cluster membership and data locations; this…

I agree we need to add new expertise to our team anyway. But I think adding bare metal expertise is easier than distributed system expertise. Etcd is indeed very interesting. I'm thinking about using it for active active replication in https://gitlab.com/gitlab-org/gitlab-ee/issues/1381

Think about it this way: your EE customers probably have the easier bare metal knowledge, but would be willing to pay for you to solve the distributed system problems for them :)

Re: Proposed server purchase for GitLab.com

#229

Post by kabdib » Mon, Dec 12, 2016, 1:45 AM UTC

Spooky23 wrote: post #114

I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…

I cannot tell you how many weird-ass storage systems I've decommissioned. Typically people go cheap, and are burned. It can be a year after deployment, or three, but one thing all the cheap stuff seems to have in common is that it fails very, very badly just when you need it very, very badly. Usually at 3 in the morning.

My philosophy today is that if the data is important at all, it's worthwhile going spendy and getting a SAN. Get a good one. I like Nimble a lot right now, but there are other good ones, too. (Don't believe crazy compression numbers or de-duplication quackery; I've told more than one SAN vendor to fuck off after they said they'd get 20:1 on our data, without doing any research on what our data was).

Have everything backed up? Great! How long until you can go live again after water drips on your storage? If you spend a week waiting for a restore, that's indistinguishable from failure. If you wait a month for replacement hardware to be shipped, you might have killed your product.

Re: Proposed server purchase for GitLab.com

#230

Post by late2part » Mon, Dec 12, 2016, 1:46 AM UTC

walrus01 wrote: post #222

late2part wrote: post #203

Earlier quoted context omitted.

Nowadays once you get past 10 racks or 50kw, you generally only pay for power - the space/cooling/etc is "free" as your limiting rate is power and the vendor's ability to move thermal. You'll likely want a chimney cabinet like the Chatsworth GlobalFrame [1]. [1] - http://www.chatsworth.com/products/cabinet-and-enclosure-sys...

This does depend a lot on geographical location. You're not going to get free racks in a major carrier exchange point in Manhattan, or downtown Seattle, or close to the urban core of San Francisco. There will definitely be a significant quantity discount, as you increase in numbers, and a lot of your cost will be power and not racks. In somewhere that is very close (OSI layer 1) topologically to a major traffic inter…

I think your point is right for 111 8th or Equinix/SV1/Great Oaks.

But for other sites with high value peering, including EQIX in Ashburn, Coresite, your limiting rate is likely to be the power, not the space or power. I.e. they'll "give" you 1 cabinet for every 15kw you buy.

So my assertion assumes you're doing large number of dense cabinets.

If one is doing a large number of dense cabinets, they almost certainly should not do it at a high value peering point, and should backhaul it. You should be able to get diverse metro dark fiber for <$5k/mo if not substantially less. Put a single cabinet (or pair for redundancy) of $2k/mo cabinets in Equinix or Fisher and off you go.