Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

71–80 of 327 posts

Re: Proposed server purchase for GitLab.com

#71

Post by gr2020 »

Think much harder about power and cooling. A few points:

1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity.

2. If you're running dual power supplies on your servers, with separate power lines coming into the rack, model what will happen if you lose one of the power lines, and all of the load switches to the other. You don't want to blow the circuit breaker on the other line, and lose the entire rack.

3. Thinking about steady state power is fine, but remember you may need to power the entire rack at full power in the worst case. Possibly from only one power feed. Make sure you have excess capacity for this.

The first time I made a significant deployment of physical servers into a colo facility, power and cooling was quite literally the last thing I thought about. I'm guessing this is true for you too, based on the number of words you wrote about power. After several years of experience, power/cooling was almost the only thing I thought about.

Re: Proposed server purchase for GitLab.com

#72

Post by toomuchtodo »

lykron wrote: post #15
sytse wrote: post #9

Earlier quoted context omitted.

Thanks for commenting! You're right that 2TB disks will have more heads per TB and therefore more IO. We want to increase performance of the 8TB drives with Bcache. If we go to smaller drives we won't be able to fit enough capacity in our common server that has only 3 drives per node. In this case we'll have to go to dedicated file servers, reducing the commonality of our setup. We're using JBOD with Ceph instead of…

It doesn't work like that. 2TB drive will always be faster than an 8TB drive. The amount of data has no effect when compared to the physical attributes of the drive. More platters will increase the response time. Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier.

> Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier.

By "Tiering", is this moving data between different drive types? Or by moving the data to different parts of the platter to optimize for rotational physics?

Re: Proposed server purchase for GitLab.com

#73

Post by toomuchtodo »

sytse wrote: post #24
neom wrote: post #10

Earlier quoted context omitted.

I'm curious to know if you think this work is within the core competency of GitLab. if so, how did you decide it was. If not, how do you realize the investment over time in something that isn't? Is the GitLab CEO here?

GitLab CEO here. Hardware and hosting are certainly not our core competencies. Hence all the questions in the blog post. And I'm sure we made some wrong assumptions on top of that. But it needs to become a core competency, so we're hiring https://about.gitlab.com/jobs/production-engineer/

> But it needs to become a core competency, so we're hiring

Impressive.

Re: Proposed server purchase for GitLab.com

#74

Post by theptip »

sytse wrote: post #36
theptip wrote: post #17

Earlier quoted context omitted.

I've been following the technical discussions around this move, and I'm wondering if you guys looked at making architectural changes to shard your data into more manageable chunks? Naively it seems like you should be able to reduce your peak filesystem iops by sharding the data at the application layer. That does introduce application complexity, but it might shake out as being less work than the operational complexi…

We have sharding on the application layer in GitLab right now https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7273 and we're using it heavily to split the load among NFS servers. Then we have to think about redundancy. The simple solution is to have an secondary NFS server and use DRBD. For the shortcomings of that read http://githubengineering.com/introducing-dgit/ The next step is introducing more granular r…

That's a fair comment RE: outsourcing, but at my company I'd bias towards bringing some distributed computing knowledge in-house rather than bringing ops expertise plus maintenance burden in-house; sounds like you're going to have to add new expertise to your team either way.

Worth investigating if you can bolt on a distributed datastore like etcd or ZooKeeper to store the cluster membership and data locations; this might not be as complex as it sounds at first. etcd gives you some very powerful primitives to work with.

(For example, etcd has the concept of expiring keys, so you can keep an up-to-date list of live nodes in your network. And you can use those same primitives to keep a strongly consistent prioritized list of repos and their backed up locations. The reconciliation component might just have to listen for node keys expiring and create and register new data copies in response.)

Re: Proposed server purchase for GitLab.com

#75

Post by skuhn »

dsr_ wrote: post #67

Z2 (no, Z is not on the list you have) You need at least two nodes that do DNS, DHCP, NTP, and other miscellaneous services that you absolutely want to have but do not seem to have mentioned. You want them to be permanently assigned, so that you never have to search for them, and you want them to fail over each service as needed, preferably both operating at once. Three nodes would be better. Consider doing some basi…

Absolutely agree. Typically I deploy about 4-6 "tools" hosts in a site. From a computational standpoint, you could make do with fewer hosts, but there are some things that I prefer to separate out:

1&2. Authoritative DNS (internal)

1&2. NTP

1&2. Caching DNS resolver

3&4. Outbound HTTP proxy (if necessary)

3&4. PXE / installer / dhcp

3&4. Local software mirror (apt / yum / etc.)

5&6. SSH jump hosts.

Or something like that.

Make sure the second host is not just in a separate chassis from the first host, but also in a separate rack.

For external authoritative DNS, don't do it yourself -- pay for someone to run that for you (Route53, ns1, etc.). For e-mail, if you can possibly not deal with it then don't -- use Mailchimp or something.

Re: Proposed server purchase for GitLab.com

#76

Post by jasoncchild »

gr2020 wrote: post #71

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the…

These are all points a competent engineer would raise :)

Re: Proposed server purchase for GitLab.com

#77

Post by thebyrd »

sytse wrote: post #24
neom wrote: post #10

Earlier quoted context omitted.

I'm curious to know if you think this work is within the core competency of GitLab. if so, how did you decide it was. If not, how do you realize the investment over time in something that isn't? Is the GitLab CEO here?

GitLab CEO here. Hardware and hosting are certainly not our core competencies. Hence all the questions in the blog post. And I'm sure we made some wrong assumptions on top of that. But it needs to become a core competency, so we're hiring https://about.gitlab.com/jobs/production-engineer/

I think you're under estimating the number of people required to run your own infrastructure. You need people who can configure networking gear, people swapping out failed NICs/Drives at the datacenter, someone managing vendor relationships, and people doing capacity planning.

I think you could probably get the IO performance you're talking about in your blog post from AWS instances or Google Cloud's local NVMe drives, but if you truly need baremetal, I'd recommend Packet or Softlayer. Don't try to run your own infrastructure or in a year you'll be: https://imgflip.com/i/1fs7it

Re: Proposed server purchase for GitLab.com

#78

Post by spydum »

gr2020 wrote: post #71

Think much harder about power and cooling. A few points: 1. Talk to your hosting providers and make sure they can support 32kW (or whatever max number you need) in a single rack, in terms of cooling. At many facilities you will have to leave empty space in the rack just to stay below their W per sq ft cooling capacity. 2. If you're running dual power supplies on your servers, with separate power lines coming into the…

fully agreed here, especially leave yourself room for when you misjudged some resource utilization (and need more). Nothing worse than having a resource crunch (cpu/mem/io) and not being able to resolve it because your rack is out of power/cooling/etc - you'll come to appreciate how easy it was in cloud just clicking the button and turning out your wallet.

Re: Proposed server purchase for GitLab.com

#79

Post by lykron »

matt4077 wrote: post #69
lykron wrote: post #15

Earlier quoted context omitted.

It doesn't work like that. 2TB drive will always be faster than an 8TB drive. The amount of data has no effect when compared to the physical attributes of the drive. More platters will increase the response time. Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier.

Though 2TB of data on an 8TB drive does mean only 1/4 as many requests hit it, right?

I'm not sure I understand your question. A 2TB/2TB disk will have the same number of requests as a 2TB/8TB disk, as they both have the same amount of data.

If you are talking physically, 2TB/8TB would theoretically be faster than a 2TB/2TB disk if the performance attributes were the same. But a 2TB HDD will have a faster average seek time than an 8TB HDD due to physical design. Any performance gains of only partially filling a drive would probably be offset by the slower overall drive.

Re: Proposed server purchase for GitLab.com

#80

Post by mritun »

dimino wrote: post #30

> We are attempting to build a fault-tolerant and performant CephFS cluster Ohhh boy. I hope this works, and look forward to hearing updates.

from CephFS website:

"Important: CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters."

I'm getting the same feeling I get when I'm watching those "Hold my beer and watch this..." videos.

From my vantage point they look to have 0 experience in building and running infrastructure... and asking advice on HN. They might ask well post a Ask Slashdot thread if they want armchair advice. Genuinely, I think they've crunched some numbers and think they can run their stuff cheaper and faster in-house... but probably underestimated the human-experience angle.

For just 10-20 physical servers, this is going to be either extremely expensive (if they hire right) or extremely painful (if they don't).

Post reply on HN