Is FreeNAS something people actually run Serious Business, at-scale production datacenters on? I've run it in my home a few times out of curiosity, and that was never my impression.
Yes. I know of several billion dollar companies that run it in a web facing production operations capacity.
Any good talks or other resources about this sort of use case that you might recommend?
We're... displeased with the current solution that we're using at work for this use case. :)
We are also a SL customer -- 4-figures of hosts with them. We have had networking problems in the past (latency and loss far higher than I would expect to see in a well-provisioned DC) and talked to them about it. It ended up being contention with another customer, it got fixed, and our network performance has been great since. I would encourage you to look at what you can get without trying to do your own colo. You'…
Thanks for the suggestion. Any idea if they can offer 40 Gbps networking?
Architecturally I would honestly wonder why you need that. Git, with its heavy reliance on immutable objects, should replicate very well. I would expect to be NIC-bound, but I would also expect it to horizontally scale reasonably well. Storage is obviously a concern -- you will have to make sure you don't end up needing a full copy of your data on every node -- but there are well known solutions to this problem that account for hot keys/objects well enough for you to get really really far. Even reaching as far back as the BigTable paper there is valuable stuff to look at and you can obviously look at Cassandra to see how the OSS world has tackled that problem.
M1: Each CPU has four memory channels, each node has 16 DIMMs (8/CPU, 2 DIMMs/channel) which means that you can use 1 DIMM per channel -> maximum memory bandwith and speed with RDIMMs. Using 64GB or 128GB LRDIMMs with E5v4 CPUs won't affect your bandwith or speed as long as you populate all the channels.[0]
Circuit breakers aren't just there for the amusement value of watching a clustered system go into split-brain mode... I.e. you would also want to be sure that your wiring was rated for 100% utilization, and that other circuit-breaker-like functions exist. Fire is an actual thing, and figuring out the best way to recharge a halon system isn't exactly what you want to be doing.
Yes. That's why they hardwire and use a breaker rated at 100%. In most jurisdiction, code requires breaker at 80% if a plug/receptacle is used. If you are hardwired you can use 100% of your capacity before tripping the breaker. You have incorrectly assumed that I suggested that you ignore breakers. When you hardwire the circuit the electrical code allows you to use a 100% breaker.
Thank you for adding this detail... what you said makes WAY more sense to me now.
Yes. I know of several billion dollar companies that run it in a web facing production operations capacity.
Any good talks or other resources about this sort of use case that you might recommend? We're... displeased with the current solution that we're using at work for this use case. :)
I've created a subreddit here [0] for discussion. Ask some questions and I'll give you the information I have that's relevant.
from CephFS website: "Important: CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters." I'm getting the same feeling I get when I'm watching those "Hold my beer and watch this..." videos. From my vantage point they look to…
We certainly don't have any experience hosting our own hardware. We're not doing this to save money, we're doing it to increase performance https://about.gitlab.com/2016/11/10/why-choose-bare-metal/
Then prototype first. Rent a server or two somewhere for 3-6 months and run a shadow first. Once you're confident that you understand all the "other 80%" stuff that is involved running your own infrastructure and don't lose data, then think about doing it yourself.
A service providers' biggest responsibilities to its customers are security, durability, availability and performance -- in that order. You guys are vastly underestimating the complexity involved in getting first 3 right.
What is the amount of money you are expecting to spend on staff vs the performance return you get?
Running your own boxes can be done, but usually at great cost and usually by blowing up your sla. Given the inexperience you have at this some other options might be politically cheaper.
If you're committed to having a robust architecture (this may not be financially viable immediately) you should study the mistakes that Github have made, e.g. https://news.ycombinator.com/item?id=11029898 Geo-redundancy seems like a luxury, until your entire site comes down due to a datacenter-level outage. (E.g. the power goes down, or someone cuts the internet lines when doing construction work on the street outsid…
There's no reason you can't have a hot cloud site, and operate in lipo mode (or even scale it up temporarily) if you lose your DC. Best of both worlds.