Goddamn, this thread is worth a few tens of thousands of dollars in consultation fees.
Yes, I think it is, thanks everyone!
Proposed server purchase for GitLab.com
271–280 of 327 posts
Re: Proposed server purchase for GitLab.com
#272Earlier quoted context omitted.
Thanks for the suggestion. Any idea if they can offer 40 Gbps networking?
Few will offer 40Gbps without charging a pretty penny, generally the jump will go from 10Gbps to 100Gbps but not until it becomes cost-effective (and not anytime soon). That said, SoftLayer does provide 20Gbps access within a private rack and 20Gbps access to the public network.
Most large scale operations are (or soon will be) deploying 25GE and/or 50GE in place of 10Gbps Ethernet. 100GE to each node is unnecessary for most workloads & more importantly it's obscenely expensive and likely to remain so for at least 3 more years.
Re: Proposed server purchase for GitLab.com
#273If the plan is still to build a huge Ceph-backed filesystem to store your git repos on, you are doomed. Redhat, if you're out there: Now would be a good time to chime in about the limits of Ceph and the reasonable size of a filesystem.
We'll have a Ceph expert review our configuration.
How about an expert in enterprise and/or cloud storage in general?
Re: Proposed server purchase for GitLab.com
#274Are you sure about the location? I would go with Frankfurt, Germany. Biggest IX in the world and if you want a "low-latency" solution for all users this is basically the middle of everything. NYC will have a worse connection to Asia and I don't want to begin with India or something. While Frankfurt is basically only 70ms away from NY and around 120ms to the west coast, while even south america should be Just as a sug…
It's a problem IMO that they are trying to pick 1 site. Geodiversity is necessary for any serious enterprise, and given that, I would suggest 1 site on the US west coast to please users in SF/LA/SEA (and most of Asia) plus 1 site somewhere in Europe (AMS, FRA, etc). Once you can afford 3 sites, add Asia or US east, depending on where the users are.
Re: Proposed server purchase for GitLab.com
#275Earlier quoted context omitted.
If you're going to spend top dollar on Arista/Cisco/EMC/NetApp, you might as well stay in the cloud. None of the clouds use any of that super-expensive gear, so if you're going for cost savings, you'll need to use the same sort of commodity gear they use. Gitlab is obviously Linux-savvy and comfortable writing automation, so things like Cumulus Linux and minimal-handholding hardware vendors shouldn't cause them any i…
But this is a couple of hundred boxes, not AWS. I've been to a Microsoft data center... the scale is infinitely larger and solutions are different well. My point isn't to knock them down. It takes cohones to be public about stuff like this. My instinct as a grumpy engineering director type is that there are holes here that need to be filled in. Putting a major product at risk to save $30k against an Arista switch isn…
cojones
Re: Proposed server purchase for GitLab.com
#276There are two major pitfalls to crowd-sourced consulting such as this. 1) Contributors have not been vetted - Some responses are based on real world experience, and some is conjecture from arm-chair quarterbacks. (A simple example would be that nobody has mentioned with any of the SuperMicro 2U Twins that you have to be cautious about the PDU models and outlet locations of 0U PDU's to not block node service/replaceme…
HN crowdsourcing is a pretty reasonable strategy for entities that cannot reliably identify & hire 1+ rockstar employee(s) and/or VARs to cover the compute, networking, storage, electrical, environmental, etc. If you can rationally evaluate the HN comments you should get pretty close to the best, cutting-edge advice. Whereas when you are small and you listen to 1 or 2 VARs and/or 1-2 internal employees you can expect, on average, to get average advice. Or advice that was excellent 2-3 years ago but is now out-of-date due to HW/SW progress that the employee/VAR is unaware of.
Re: Proposed server purchase for GitLab.com
#277Earlier quoted context omitted.
I've found that being considered a 'competent engineer' merely means 'never too arrogant to learn'. I didn't know half of the stuff in grandparents' post.
I was attempting to confirm that these are all points are important. Being too arrogant to learn and being in incompetent are two entirely different things.
Sorry, I misunderstood. To me, your response came across as a "well, duh!".
Re: Proposed server purchase for GitLab.com
#278My initial gut feeling is that you are moving out of the cloud for the wrong reasons. Any performance gain you get with bare metal will be erased with the complexity of running a hybrid environment, namely moving data back and forth between your datacenter and the cloud, and also the mental overhead of programming to that model.
You also now need to gain internal expertise in networking, security, datacenter operations, and people who can rack and stack well.
Re: Proposed server purchase for GitLab.com
#279Earlier quoted context omitted.
> As someone who has run large scale Ceph before (though not CephFS, thankfully), it's not easy to run at scale. We had a team of 5 software engineers as well as an Ops and Hardware team, and we had to do a lot to get it stable. It's not as easy as installing it and walking away. Can you give some examples of the problems you ran into?
Definitely! Not going to be an exhaustive list, but I can talk about some of the bigger pieces of work. Something that always seemed to cause nagging issues was that we wanted our cluster to have data encryption at rest. Ceph does not support this out of the box, which means that you need to use dmcrypt on top of your partitions, and present those encrypted partitions to Ceph. This requires some work to make sure tha…
> This requires some work to make sure that decrypt keys are setup properly, and that the machine can reboot automatically and remount the proper partitions.
I've always wondered how automatic reboots are handled with filesystem encryption.
What's the process that happens at reboot?
Where is the key stored?
How is it accessed automatically?
Re: Proposed server purchase for GitLab.com
#280Earlier quoted context omitted.
I'm curious to know if you think this work is within the core competency of GitLab. if so, how did you decide it was. If not, how do you realize the investment over time in something that isn't? Is the GitLab CEO here?
GitLab CEO here. Hardware and hosting are certainly not our core competencies. Hence all the questions in the blog post. And I'm sure we made some wrong assumptions on top of that. But it needs to become a core competency, so we're hiring https://about.gitlab.com/jobs/production-engineer/
You are ruling out providers like OVH because lack of sufficient SLA but then you are replacing it with your own solution that will have no SLA.
Lower your SLA requirements and go with multiple providers like OVH. Make your site work at multi datacenters. At the end of the day your users will be much happier.
My 2 cents.