Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

11–20 of 327 posts

Re: Proposed server purchase for GitLab.com

#11

Post by olalonde »

deadbunny wrote: post #7

Yeah, link aggregation doesn't work how they think it does. And not having a separate network for Ceph is going to bite them in the arse. GitLab is fine software but fuck me, they need to hire someone with actual ops experience (based on this post and their previous "we tried running a clustered file system in the cloud and for some reason it ran like shit" post).

Seems this is one of the goals of the article: "We're hiring producton engineers and if you're spotting mistakes in this post we would love to talk to you (and if you didn't spot many mistakes but think you can help us we also want to talk to you)".

Re: Proposed server purchase for GitLab.com

#12

Post by octo_t »

It seems based on this post that all the hardware is going to end up in one datacenter. that seems like a big risk if that datacenter ends up having issues.

Perhaps look into how to have the nodes split between 2 or 3 different datacenters?

Re: Proposed server purchase for GitLab.com

#13

Post by sytse »

nik736 wrote: post #8

Are you sure about the location? I would go with Frankfurt, Germany. Biggest IX in the world and if you want a "low-latency" solution for all users this is basically the middle of everything. NYC will have a worse connection to Asia and I don't want to begin with India or something. While Frankfurt is basically only 70ms away from NY and around 120ms to the west coast, while even south america should be Just as a sug…

Frankfurt surely has a great IX. Even if Frankfurt would make everyone better off (which I'm not sure about) there is another problem. People in the US are used to lower latencies because most SaaS services are hosted there.

Re: Proposed server purchase for GitLab.com

#14

Post by gsylvie »

I have a Cisco c220 m4 (http://www.cisco.com/c/dam/en/us/products/collateral/servers...). I was surprised at the complexity of the configuration options. For example, it has 24 memory slots, but if you fill them memory speed is 2133mhz, whereas if you only use 16 memory slots, max memory speed is 2400mhz.

I've ordered a pair of Intel 750 Series 1.2TB NVMe SSD's for it, can't wait to try that out. Still waiting for the SSD drives to arrive.

Re: Proposed server purchase for GitLab.com

#15

Post by lykron »

sytse wrote: post #9
lykron wrote: post #4

If you are looking for performance, do not get the 8TB drives. In my experience, drives above 5TB do not have good response times. I don't have hard numbers, but I built a 10 disk RAID6 array with 5TB disks and 2TB disks and the 2TB disks were a lot more responsive. From my own personal experience, I would go with a PCIe SSD cache/write buffer, and then a primary SSD tier and a HDD tier. Storage, as it seems you guys…

Thanks for commenting! You're right that 2TB disks will have more heads per TB and therefore more IO. We want to increase performance of the 8TB drives with Bcache. If we go to smaller drives we won't be able to fit enough capacity in our common server that has only 3 drives per node. In this case we'll have to go to dedicated file servers, reducing the commonality of our setup. We're using JBOD with Ceph instead of…

It doesn't work like that. 2TB drive will always be faster than an 8TB drive. The amount of data has no effect when compared to the physical attributes of the drive. More platters will increase the response time.

Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier.

Re: Proposed server purchase for GitLab.com

#16

Post by deadbunny »

olalonde wrote: post #11
deadbunny wrote: post #7

Yeah, link aggregation doesn't work how they think it does. And not having a separate network for Ceph is going to bite them in the arse. GitLab is fine software but fuck me, they need to hire someone with actual ops experience (based on this post and their previous "we tried running a clustered file system in the cloud and for some reason it ran like shit" post).

Seems this is one of the goals of the article: "We're hiring producton engineers and if you're spotting mistakes in this post we would love to talk to you (and if you didn't spot many mistakes but think you can help us we also want to talk to you)".

My bad, I didn't make it that far down the post before hitting rant mode.

Re: Proposed server purchase for GitLab.com

#17

Post by theptip »

sytse wrote: post #5
AstroJetson wrote: post #3

Love how they did some number crunching and decided that rent vs own, own won. I think that if more places looked they would find that out also. There must be a margin in it since the big players are making money at it. I'm interested to see what they end up with in the end.

Thanks! The decision to move to metal was because of performance problems https://about.gitlab.com/2016/11/10/why-choose-bare-metal/ It is nice that we'll save on costs but we anticipate a lot of extra complexity that will slow us down. So if it wasn't needed we would have stayed in the cloud. But it is interesting that both our competitors (GitHub.com and BitBucket.org) also moved to metal.

I've been following the technical discussions around this move, and I'm wondering if you guys looked at making architectural changes to shard your data into more manageable chunks?

Naively it seems like you should be able to reduce your peak filesystem iops by sharding the data at the application layer. That does introduce application complexity, but it might shake out as being less work than the operational complexity of running my own metal.

Of course, easier said than done -- I just didn't spot any discussion of this option, and it seemed like the design choice of having one filesystem served by Ceph was taken for granted.

Re: Proposed server purchase for GitLab.com

#18

Post by sytse »

octo_t wrote: post #12

It seems based on this post that all the hardware is going to end up in one datacenter. that seems like a big risk if that datacenter ends up having issues. Perhaps look into how to have the nodes split between 2 or 3 different datacenters?

We've considered having part of the servers in a different location. But it is a big risk to see if Ceph can handle the latency. We're also considering using GitLab Geo to have a secondary installation. It seems that data centers can have intermittent network or power issues but they are less likely to go down for days (for example Backblaze is in one DC). At some point we'll likely have multiple datacenters but our first focus is to make GitLab.com fast as soon as possible.

Re: Proposed server purchase for GitLab.com

#19

Post by alfalfasprout »

lykron wrote: post #4

If you are looking for performance, do not get the 8TB drives. In my experience, drives above 5TB do not have good response times. I don't have hard numbers, but I built a 10 disk RAID6 array with 5TB disks and 2TB disks and the 2TB disks were a lot more responsive. From my own personal experience, I would go with a PCIe SSD cache/write buffer, and then a primary SSD tier and a HDD tier. Storage, as it seems you guys…

So, the 8TB drives are fine if you're doing sequential writes, mainly doing write-only workloads, or have a massive buffer in front.

In my experience, the PCIE drives they mention (the DC P3700) are incredible. They're blazing fast (though the 3D X-Point stuff is obviously faster), have big supercaps (in case of power failure), and really low latency for an SSD (150 microseconds or so). They're a pretty suitable alternative to in-memory logbuffers where latency is less crucial. Much cheaper than the equivalent memory too. Having a few of these in front of a RAID 10 made up of 8TB drives will work just fine.

FWIW your experience with a RAID 6 of big disks is unsurprising. Raid 6 is almost always a bad choice nowadays since disks are so big. Terrible recovery times and middling performance.

Re: Proposed server purchase for GitLab.com

#20

Post by samstave »

sytse wrote: post #13
nik736 wrote: post #8

Are you sure about the location? I would go with Frankfurt, Germany. Biggest IX in the world and if you want a "low-latency" solution for all users this is basically the middle of everything. NYC will have a worse connection to Asia and I don't want to begin with India or something. While Frankfurt is basically only 70ms away from NY and around 120ms to the west coast, while even south america should be Just as a sug…

Frankfurt surely has a great IX. Even if Frankfurt would make everyone better off (which I'm not sure about) there is another problem. People in the US are used to lower latencies because most SaaS services are hosted there.

I have less of a concern on latency than I do with who has taps in the lines...

I would assume that the .de lines are saturated with 5 eyes...

Post reply on HN