Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

Post reply on HN
Print view View original

171–180 of 327 posts

Re: Proposed server purchase for GitLab.com

#171

Post by sytse »

mritun wrote: post #80
dimino wrote: post #30

> We are attempting to build a fault-tolerant and performant CephFS cluster Ohhh boy. I hope this works, and look forward to hearing updates.

from CephFS website: "Important: CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters." I'm getting the same feeling I get when I'm watching those "Hold my beer and watch this..." videos. From my vantage point they look to…

We certainly don't have any experience hosting our own hardware.

We're not doing this to save money, we're doing it to increase performance https://about.gitlab.com/2016/11/10/why-choose-bare-metal/

Re: Proposed server purchase for GitLab.com

#172

Post by ninkendo »

manacit wrote: post #144
Spooky23 wrote: post #114

I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…

I'm happy to see this - I could not agree more with these points. I think they are coming at this problem from the wrong perspective - instead of growing from virtual servers to their own dedicated hardware to get better CephFS performance, they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that t…

I don't see why it wouldn't be difficult to do a git implementation that replaces all the FS syscalls with calls to S3 or native ceph or some other object store. If all they're using NFS for is to store git, it seems like a big win to put in the up front engineering cost.

I mean, especially because git's whole model of object storage is content-addressable and immutable, it looks like it's a prime use for generic object storage.

Re: Proposed server purchase for GitLab.com

#173

Post by StreamBright »

Spooky23 wrote: post #114

I'm a cranky old person now, I think this is a crazy approach to take and I would be having a very challenging conversation with the engineer pitching this to me. My underlying assumption is that this is a production service with customers depending on it. 1. Don't fuck with networking. Do you have experience operating same or similar workloads on your super micro sdn? Will the CEO of your super micro VAR pickup his…

Couldn't agree more. It seems Gitlab is severely underestimating the cost of having similar infrastructure on bare metal as their AWS infra. I would probably start to re-architect their software and replace Ceph with something better, probably with S3. Much easier to scale up and operate. Also their Ruby stack has a lot of optimisation potential. Starting to optimise on the other end (hardware, networking, etc.) is a much harder job, starting with staffing questions. AWS, Google and MSFT has the best datacenter engineers you can find and there is a huge amount of effort went into engineering their datacenters. Not to mention your leverage you have being a small startup vs a cloud vendor when talking to HW vendors. Anyways, in few years we gonna know if they managed to do this successfully.

Re: Proposed server purchase for GitLab.com

#174

Post by sulam »

i336_ wrote: post #162

Here's a crazy but interesting suggestion: > Backup > ...Even with RAID overhead it should be possible to have 480TB of usable storage (66%). Quoting https://code.facebook.com/posts/1433093613662262/-under-the-... : > Fortunately, with a technique called erasure coding, we can. Reed-Solomon error correction codes are a popular and highly effective method of breaking up data into small pieces and being able to easily…

Any halfway experienced storage engineer is fluent in ECC. You don't need secret sauce from Facebook. That said, because of the first statement, a lot of today's storage solutions will use ECC on the backend if they present you with a logical FS. So you may not (should not?) need to reinvent this wheel. At Facebook's scale this absolutely makes sense, but they're not particularly breaking new ground here. Look at RAID 6 if you need more evidence.

Re: Proposed server purchase for GitLab.com

#175

Post by revelation »

MichaelGG wrote: post #137
wmf wrote: post #83

I have built out a few racks of Supermicro twins. In general I would suggest hiring your ops people first and then letting them buy what they are comfortable with. C2: The Dell equivalent is C6320. CPU: Calculate the price/performance of the server , not the processor alone. This may lead you towards fewer nodes with 14-core or 18-core CPUs. Disk: I would use 2.5" PMR (there is a different chassis that gives 6x2.5" p…

Can you elaborate on what deficiencies 10GBase-T has in server applications?

I figure it's simply the availability of datacenter level switches (and other components).

Re: Proposed server purchase for GitLab.com

#176

Post by sytse »

lykron wrote: post #86
toomuchtodo wrote: post #72

Earlier quoted context omitted.

> Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier. By "Tiering", is this moving data between different drive types? Or by moving the data to different parts of the platter to optimize for rotational physics?

By Tiering, I'm talking about moving blocks of data from slower to faster or visa verse. If I have 10TB of 10k IOPS storage and 100TB of 1k IOPS storage in a tiered setup, data that is frequently accessed would reside in the 10k IOPS tier while less frequently accessed data would be in the 1k tier. In this case, the blocks of popular repositories would be stored in SSD, while the blocks of your side project that you…

That is pretty awesome. Should the SSD's for the fast storage be on the OSD nodes that also have the HDD's or should it be separate OSD nodes?

Re: Proposed server purchase for GitLab.com

#177

Post by stonogo »

Is there a reason you don't just put out an RFP and purchase a complete supported solution from a VAR or OEM vendor?

I'm not sure you have the in-house expertise to maintain a production service of this kind. That's not an attack; this has not been your focus in the past, so it might be wise to have a third party provide some assistance and support as an intermediate step toward doing everything in-house.

Re: Proposed server purchase for GitLab.com

#178

Post by manacit »

ninkendo wrote: post #172
manacit wrote: post #144

Earlier quoted context omitted.

I'm happy to see this - I could not agree more with these points. I think they are coming at this problem from the wrong perspective - instead of growing from virtual servers to their own dedicated hardware to get better CephFS performance, they should take a hard look at their application and see if they can architect it in a way that does not require a complex distributed filesystem to present a single mount that t…

I don't see why it wouldn't be difficult to do a git implementation that replaces all the FS syscalls with calls to S3 or native ceph or some other object store. If all they're using NFS for is to store git, it seems like a big win to put in the up front engineering cost. I mean, especially because git's whole model of object storage is content-addressable and immutable, it looks like it's a prime use for generic obj…

This is precisely how I would recommend doing this if I were to be asked. Because git is content-addressable, it lends itself very well to being stored in an object storage system (as long as it has the right consistency guarantees). Instead of using CephFS, they could use Ceph's rados gateway which would allow them to abstract the storage engine away to working with any object storage system.

Latency and consistency would be my concerns - S3 does not quite have the right semantics for some of this, so you'd have to build a small shim on top to work around this. Ceph's rados doesn't even have these problems, so it is quite a good contender.

Re: Proposed server purchase for GitLab.com

#179

Post by sytse »

skuhn wrote: post #49

For server hardware, the Supermicro 2U Twins are a reasonable choice, but I prefer their 4U FatTwin chassis. The engineering quality is a little better IMO, and the cost increase isn't too big. Absolutely do not buy their 1U Twin systems, they are hot garbage. The FatTwin chassis has similar density, and can support either 1U half width or 2U half width systems in a particular chassis. Typically I use 1U's for app /…

Thanks! But I don't see the 2630v4 in that sheet, only older versions.
Post reply on HN