Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

291–300 of 327 posts

Re: Proposed server purchase for GitLab.com

#291

Post by late2part »

mohctp wrote: post #243
late2part wrote: post #210

Earlier quoted context omitted.

note that you can get around the 80% breaker limit by having your DC hardwire the power, if you have enough scale to have them do this for you.

This really is only a concern when you are paying a monthly recurring charge (MRC) by the breaker amp with many power drops. For a deployment of this scale it should be metered power (For example 1 (or more) 3phase a+b drops to each cabinet) where you only pay a Non-Recurring setup Charge (NRC) and then the MRC is based on actual power draw. 3phase also means fewer physical PDU's (uses less space), but more physical…

Most DC PDUs will stagger your outlet power on to stagger the initially large power draws.

Re: Proposed server purchase for GitLab.com

#293

Post by Dylan16807 »

deadbunny wrote: post #7

Yeah, link aggregation doesn't work how they think it does. And not having a separate network for Ceph is going to bite them in the arse. GitLab is fine software but fuck me, they need to hire someone with actual ops experience (based on this post and their previous "we tried running a clustered file system in the cloud and for some reason it ran like shit" post).

They said very little about how they think link aggregation works. Just that they can send packets on both and continue working with only one port. That's basically the definition of link aggregation. So what's wrong about the post?

Re: Proposed server purchase for GitLab.com

#294

Post by Dylan16807 »

lykron wrote: post #79
matt4077 wrote: post #69

Earlier quoted context omitted.

Though 2TB of data on an 8TB drive does mean only 1/4 as many requests hit it, right?

I'm not sure I understand your question. A 2TB/2TB disk will have the same number of requests as a 2TB/8TB disk, as they both have the same amount of data. If you are talking physically, 2TB/8TB would theoretically be faster than a 2TB/2TB disk if the performance attributes were the same. But a 2TB HDD will have a faster average seek time than an 8TB HDD due to physical design. Any performance gains of only partially…

> a 2TB HDD will have a faster average seek time than an 8TB HDD due to physical design. Any performance gains of only partially filling a drive would probably be offset by the slower overall drive.

I'm skeptical here. So your minimum seek time goes up on the 8TB because it's harder to align. But your maximum seek time should drop tremendously because the drive arm only has to move along the outer fifth of the drive. And your throughput should be great because of the doubled linear density combined with using only the outer edge.

Re: Proposed server purchase for GitLab.com

#295

Post by Dylan16807 »

dsr_ wrote: post #56
sytse wrote: post #18

Earlier quoted context omitted.

We've considered having part of the servers in a different location. But it is a big risk to see if Ceph can handle the latency. We're also considering using GitLab Geo to have a secondary installation. It seems that data centers can have intermittent network or power issues but they are less likely to go down for days (for example Backblaze is in one DC). At some point we'll likely have multiple datacenters but our…

Do you have a disaster recovery plan that starts with "A meteor has destroyed our primary data center."? I do; that's my default scenario. If you can survive that, you can survive all sorts of smaller issues like network congestion, data center power problems, grid power problems, and zombie plagues (or flu, which is more likely.)

Depends on what you mean by 'survive'. I'd call a backup in google nearline sufficient for the meteor scenario, but that's going to be very slow and unpleasant to depend on for milder problems.

Re: Proposed server purchase for GitLab.com

#296

Post by lykron »

Dylan16807 wrote: post #294
lykron wrote: post #79

Earlier quoted context omitted.

I'm not sure I understand your question. A 2TB/2TB disk will have the same number of requests as a 2TB/8TB disk, as they both have the same amount of data. If you are talking physically, 2TB/8TB would theoretically be faster than a 2TB/2TB disk if the performance attributes were the same. But a 2TB HDD will have a faster average seek time than an 8TB HDD due to physical design. Any performance gains of only partially…

> a 2TB HDD will have a faster average seek time than an 8TB HDD due to physical design. Any performance gains of only partially filling a drive would probably be offset by the slower overall drive. I'm skeptical here. So your minimum seek time goes up on the 8TB because it's harder to align. But your maximum seek time should drop tremendously because the drive arm only has to move along the outer fifth of the drive.…

I'm not saying that you wouldn't see any performance gain. 2TB on the outer track will be faster than 8TBs on the same 8TB disk, but I'm saying any gains will be lost due the dense nature of the drives.

A quick google search shows that there are marginal gains on the outer track vs the inner, but that is only on sequential workloads. For something like GitLab, the workloads would be anything but.

http://superuser.com/questions/643013/are-partitions-to-the-...

https://www.pythian.com/blog/hard-drive-inner-or-outer/

Re: Proposed server purchase for GitLab.com

#297

Post by lykron »

campervan wrote: post #247
lykron wrote: post #197

Earlier quoted context omitted.

I hope to god you are not using Kubernetes for your databases.

Why not? Google Cloud SQL runs on Kubernetes.

Docker, and Kubernetes is still a rapidly evolving technology and there are too many horror stories of it crashing and burning in production for it to be worth it, in my view. Databases are not immutable, and even though Docker has persistent volumes it is still new and evolving. If the database breaks in some way, I don't want to have to worry about the new container having a slightly different patch level and that messing up the database. I don't want to have to worry about rebuilding my database cluster to upgrade it.

Google might do some really cool shit with Kubernetes, but they are Google. I don't think using them as a good example for anything but infrastructure on a massive scale is correct. They are years ahead of everyone else, and if they are doing something in production, they have been testing it for years. If shit hits the fan, they have thousands of employees to throw at the problem. GitLab is building a team, and therefore does not have the experience to know these systems inside and out. In my view, using Docker/Kubernetes is adding unnecessary complexity to the database fabric for minimal tradeoffs.

And considering that they are speaking out dedicated database servers, it makes no sense to add a unneeded layer of abstraction when in all likelihood the container will be bound to a node.

Re: Proposed server purchase for GitLab.com

#298

Post by illumin8 »

sytse wrote: post #176
lykron wrote: post #86

Earlier quoted context omitted.

By Tiering, I'm talking about moving blocks of data from slower to faster or visa verse. If I have 10TB of 10k IOPS storage and 100TB of 1k IOPS storage in a tiered setup, data that is frequently accessed would reside in the 10k IOPS tier while less frequently accessed data would be in the 1k tier. In this case, the blocks of popular repositories would be stored in SSD, while the blocks of your side project that you…

That is pretty awesome. Should the SSD's for the fast storage be on the OSD nodes that also have the HDD's or should it be separate OSD nodes?

The fact that you're asking this question on Hacker News leaves little doubt in my mind that you and your team are not prepared for this (running bare metal).

I read the entire article, and while you talked about having a backup system (in a single server, no less!) that can restore your dataset in a reasonable amount of time, you have no capability for disaster recovery. What happens when a plumbing leak in the datacenter causes water to dump into your server racks? How long did it take you to acquire servers, build the racks and get them ready for you to host customer data? Can your business withstand that amount of downtime (days, weeks, months?) and still operate?

These questions are the ones you need to be asking.

In other words, double your budget because you'll need a DR site in another datacenter.

Re: Proposed server purchase for GitLab.com

#299

Post by cmn »

charliesome wrote: post #193
ninkendo wrote: post #172

Earlier quoted context omitted.

I don't see why it wouldn't be difficult to do a git implementation that replaces all the FS syscalls with calls to S3 or native ceph or some other object store. If all they're using NFS for is to store git, it seems like a big win to put in the up front engineering cost. I mean, especially because git's whole model of object storage is content-addressable and immutable, it looks like it's a prime use for generic obj…

Latency is an issue. Especially when traversing history in operations like log or blame, it is important to have an extremely low latency object store. Usually this means a local disk.

Yuup. Latency is a huge issue for Git. Even hosting Git at non-trivial scales on EBS is a challenge. S3 for individual objects is going to take forever to do even the simplest operations.

And considering the usual compressed size of commits and many text files, you're going to have more HTTP header traffic than actual data if you want to do something like a rev-list.

Re: Proposed server purchase for GitLab.com

#300

Post by mohctp »

late2part wrote: post #258
mohctp wrote: post #243

Earlier quoted context omitted.

This really is only a concern when you are paying a monthly recurring charge (MRC) by the breaker amp with many power drops. For a deployment of this scale it should be metered power (For example 1 (or more) 3phase a+b drops to each cabinet) where you only pay a Non-Recurring setup Charge (NRC) and then the MRC is based on actual power draw. 3phase also means fewer physical PDU's (uses less space), but more physical…

Not really, although I agree with your reasoning. The other is issue is capex. When I deploy 240kW pods, if I use 80% breakers, I have to deploy 25% more PDUs than if I have 100% breakers. Since my cabinet number is usually evenly divisible by N*PDUs, this impacts overall capital.

We are talking about 1 to 2 cab density here so capex doesn't carry that much weight.

Having a little headroom on your power circuits is also incredibility important, and not every facility will sell 100% rated breakers. It may make more sense to be in a facility with 80% rated breakers than 100%, even with the added capex of an extra PDU or two.

Goes back to my previous comment. What is important to you, at the pod / multiple pod level, isn't as important to the 1-2 cab deployment.

Post reply on HN