Live data from Hacker News

Proposed server purchase for GitLab.com

about.gitlab.com

61–70 of 327 posts

Re: Proposed server purchase for GitLab.com

#61
Z1: the word "monitoring" does not appear in this document.

You will need to monitor: - ping - latency - temperatures - cpu utilization - ram utilization - disk utilization - disk health - context switches - IP addresses assigned and reaching expected MACs - appropriate ports open and listening - appropriate responses - time to execute queries - processes running - process health - at least something for every bit of infrastructure

once you collect that information, you need to record it, graph it, display it, recognize non-normal conditions, alert on those, page for appropriate alerts, and figure out who answers the pagers and when.

Re: Proposed server purchase for GitLab.com

#62
post #8

Are you sure about the location? I would go with Frankfurt, Germany. Biggest IX in the world and if you want a "low-latency" solution for all users this is basically the middle of everything. NYC will have a worse connection to Asia and I don't want to begin with India or something. While Frankfurt is basically only 70ms away from NY and around 120ms to the west coast, while even south america should be Just as a sug…

If you can only deploy one place in the world, east coast US is generally the right fit (if you have typically distributed user traffic, of course). You can cover western Europe and the major population centers of eastern North America, and still have reasonable performance in western North America as well. While it's increasingly less true over time, keep in mind that this is the place that other countries all prioritized building connectivity to, because historically this is where early commercial websites lived.

Of course there's the issue of potential legalities for your business, but don't kid yourself that you're safe from prying eyes by deploying in a particular country.

The next step (and particularly important for a business like GitLab) is to land a second site in either western Europe or west coast US. Honestly you should be thinking about this right away, and look to sign leases on both spaces simultaneously with a 3-mo delay built in for the second site. This should help you negotiate price as well, if you're able to go with the same dc provider, but be aware that that itself is a single point of failure as well. Make absolutely sure you negotiate your MSA to the n-th degree, get a good SLA, etc. You can still get burned, but do your legal due diligence now because you won't have a chance to change terms later.

Then continue to optimize by having multiple sites per-region (so that failover doesn't involve a big performance hit), adding APAC / AUNZ regions, and so forth. For a service like GitLab, I wouldn't think that time to sync the repo is hyper important, but responsiveness of the web interface is fairly key. So that may lead to a hub/spoke design where there are a few larger sites storing the bulk of the data and more small sites to handle metadata and such to present the web views.

That's all years down the road though. For a first pass, I can't see a problem with northern Virginia as the first site.

Re: Proposed server purchase for GitLab.com

#63
Since hosting git repositories is core to your business, you should take the time to do it right (https://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/fi...) instead of using vanilla git and relying on a magic filesystems and vertical scaling to solve your issues.

Re: Proposed server purchase for GitLab.com

#64
post #34

If you're committed to having a robust architecture (this may not be financially viable immediately) you should study the mistakes that Github have made, e.g. https://news.ycombinator.com/item?id=11029898 Geo-redundancy seems like a luxury, until your entire site comes down due to a datacenter-level outage. (E.g. the power goes down, or someone cuts the internet lines when doing construction work on the street outsid…

This is definitely what perked my ears up when they mentioned the US East Coast, especially when you consider the risk that a natural disaster might take out the facility.

Re: Proposed server purchase for GitLab.com

#65
post #55
post #49

For server hardware, the Supermicro 2U Twins are a reasonable choice, but I prefer their 4U FatTwin chassis. The engineering quality is a little better IMO, and the cost increase isn't too big. Absolutely do not buy their 1U Twin systems, they are hot garbage. The FatTwin chassis has similar density, and can support either 1U half width or 2U half width systems in a particular chassis. Typically I use 1U's for app /…

And if you want to upgrade later, I think you should estimate about ~3 years before 128GB DDR4 LR-DIMMs are cost effective, right?

It's hard to say with absolute certainty, but I think 2-3 years is a reasonable guess. 64GB DIMMs have only recently become semi-reasonable, and I still use a lot of 16GB or 32GB DIMMs on smaller deployments.

Basically whatever the top-of-the-line DIMM option may be (and this applies for CPUs and HDDs and other stuff too), you want to avoid being in a situation where you HAVE to use it. Vendors price these parts accordingly: you pay a premium for top-of-the-line because you must have it. If you can avoid that, do so.

Re: Proposed server purchase for GitLab.com

#66
A bit confused why you are trying to roll your own storage solution. Ceph is great for a lot of applications, but I am not sure it really fits the bill for what you describe. Especially when you indicate you are going to use spinning disk behind it. Have you looked at any of the storage arrays on the market? Your TCO is likely to be much lower and your performance/resiliency much higher if you buy something with $100's of millions of R&D behind it rather than all the hours and costs of rolling your own. Not saying its impossible to make it work, but it just sounds like something that will be a PITA going forward.

Re: Proposed server purchase for GitLab.com

#67
Z2 (no, Z is not on the list you have)

You need at least two nodes that do DNS, DHCP, NTP, and other miscellaneous services that you absolutely want to have but do not seem to have mentioned. You want them to be permanently assigned, so that you never have to search for them, and you want them to fail over each service as needed, preferably both operating at once. Three nodes would be better. Consider doing some basic monitoring on these nodes, too, as a backup to your main monitoring system.

Re: Proposed server purchase for GitLab.com

#68
Kudos to gitlab for being so open about this. As an outsider find it very interesting to observe and learn from this transition.

It's refereshing too see that a few quite important players including i.e. dropbox are moving away from having everything in public clouds in contrast to others such as Netflix that go all in. Looks like on premise is not dead yet.

Good luck!

Re: Proposed server purchase for GitLab.com

#69
post #15
post #9

Earlier quoted context omitted.

Thanks for commenting! You're right that 2TB disks will have more heads per TB and therefore more IO. We want to increase performance of the 8TB drives with Bcache. If we go to smaller drives we won't be able to fit enough capacity in our common server that has only 3 drives per node. In this case we'll have to go to dedicated file servers, reducing the commonality of our setup. We're using JBOD with Ceph instead of…

It doesn't work like that. 2TB drive will always be faster than an 8TB drive. The amount of data has no effect when compared to the physical attributes of the drive. More platters will increase the response time. Ceph seems to offer Tiering, which would move frequently accessed data into a faster tier while the infrequent data to a slower tier.

Though 2TB of data on an 8TB drive does mean only 1/4 as many requests hit it, right?

Re: Proposed server purchase for GitLab.com

#70
It'd be interesting to see more application-level solutions to scale rather than just adding hardware. Like extending git so objects can be fetched via a remote object-store rather than dealing with a locally mounted POSIX file system. This would allow you to use native cloud object stores and might simplify your latency requirements to the point where you could consider staying in the cloud.

This definitely increases software complexity but going the other way increases other complexities (ops, capex, capacity).

Post reply on HN