Servers:
C1: Have a look at the FatTwin Line for more Disks per U. More PCIe Slots too.
Disks
D3: Check the measurements, having it not fit is painful
D4: More, smaller drives. Make sure you go PMR not SMR if you do
go for 8TB
Network
N1: The "SDN" aspect of the supermicro one is not really
any different than any other. Look at https://bm-switch.com/
and get one that supports Cumulus Linux. Buy one with an
x86 CPU. If you want to do "SDN" things or run custom monitoring, not dealing with PPC is great.
N3: Probably not needed, but not terribly expensive if it provides benefit.
N4: see N1, no.
N5: Cheap 1G switch that supports cumulus, x86, probably broadcom \
helix4
Networking General:
- I wouldn't advise using the 10GbE Copper --Go for 25GbE with DAC, it's basically the same price, Mellanox NICs are small/cheap
- Transit is cheap, you can get 500Mbps on a 10G port for $325/mo from Cogent
- If your bandwidth needs to scale up, data center locations matter more than you think
25GbE adapter -- minimal additional cost for 2.5x the perf, lower latency as well:
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=2...
a 32 port 100GbE switch is about $7000-12,000 -- you can break that down to 128x 25GbE, and use the 25GbE ports running at 10GbE mode for your carrier uplinks. Could even do
100GbE to your Ceph nodes if you wanted, but be aware of PCIe bottlenecks -- x8 is about 64Gbps, x16 will do 125Gbps. Dual port 40G on x8 or dual port 100G on x16 will not provide more than those numbers.
Consider Supermicro NVMe servers (Ultra series) for DBs, and 2.5in NVMe SSDs instead of PCIe.
Rack:
Don't assume 45U
40-48 is common. Consider buying 2 racks.
Power:
19kW seems high for a single rack, you will need a good
datacenter to support that density. Density costs money
more racks is cheaper generally.
208v * 30A = 5000W usable per feed, unless they are talking about
208v 3-phase, which gets you 8600W usable per feed, which again
is only 17kW and you need 18-20kW.
helpful reference: http://www.raritan.com/blog/detail/3-phase-208v-power-strips...
You can only use 80% of your power provided.
You also need Rack PDUs, higher density PDUs cost more money
consider buying port-switchable PDUs. Raritan makes good ones.
Ask supermicro (or your reseller) for a "Power Sheet" it will tell you almost
exactly how much power your server will use. I've had good luck with ThinkMate
Hosting
H1: Yes, too many to mention
H2: Do it yourselves
H4: yes
Hosting general: Cross connects cost money, a number of facilities offer free xconnects, this can add up.
Other notes:
- You want a small toolbox in the data center
- Buy more cables than you think you'll need
- You'll always forget something
- There are a number of companies that will lease you servers for pretty decent rates