Proposed server purchase for GitLab.com

Cumulus isn't really SDN so I'm not sure what you're saying there.

Traditional networking is fine but it's totally different than Linux so you need dedicated netops people to manage it. And Cisco is the most expensive traditional vendor.

Where will my DC be? What kind of DC is it? What services do they provide? How long do I want it to take for an employee to get there, whether or not they have 24/7 remote hands? What kind of power resiliency do they provide? What will power cost? What kind of power do they provide per cage and rack? How will their uplinks affect my traffic needs? Etc etc.

Cooling I didn't deal with directly, but suffice to say you will always need more cooling, and its efficacy will determine if your hardware stays alive or not. I've seen 11 foot racks with only 6 feet of hardware because they simply couldn't cool the racks at full height. Learn how to look for properly designed hot/cold racks and keep your racks well organized to make cooling efficient.

Power is pretty obvious, except that it isn't. Eventually you will draw too much power and you'll need to shunt machines into different racks and monitor your power trends. So one of the things to consider, besides dual drops, is how many extra racks do I have for when I need to spread out my power OR cooling into new racks? Will they make me use a rack on the other side of the building, or am I going to pay for some in reserve next to the current ones? Get PDUs that aren't a pain in the ass to automate (APC sucks balls).

Network: i'm not a neteng, don't listen to me, but obviously it should be managed with nice fat switching fabric bandwidth, good forwarding rate and big uplink module support. 48-port switches don't always have the same bandwidth ratios as 24-port switches, and uplinks are much easier to manage on a 24-port than a 48.

Hardware: you don't seem to need anything special, so you need to determine if a support contract is necessary, and if not, buy the cheapest pieces of shit you can and then rely on remote hands or a local employee to change out broken shit all the time. If space, power, cooling are at a premium, a blade chassis can be handy. But if you can spare the space, power, and cooling, 0.5U and 1U shitboxes are fine for most purposes. Don't get wrapped up in the details unless your application design requires specific hardware performance guarantees.

Looked at iSCSI SANs? Could make WAN sync easier, reduce overhead from NFS, but probably depends on how well your OS supports it and the features of the SAN. Oh, and an OOB terminal server can be a godsend when combined with a good PDU.

Go find all the industry datacenter design papers out there (there are tons) to bone up on the design considerations. Remember that you can always replace machines, but you can't replace rack, cooling or power design.

A system at idle vs full CPU vs full cpu + all disk will produce very different measurements.

Also keep in mind 80% derating - many electrical codes will state that an X amp circuit should only be used at 0.8X on a continuous basis (and the circuit breakers will be sized accordingly).

Disclaimer: Prometheus co-founder and just started as a Prometheus contractor for GitLab.

EDIT: ah, didn't reload the page to see that sytse had already responded with this :)