Fun piece, thanks to the author. But for vicarious thrills like this, more pictures are always appreciated!
Many colos disallow photography.
Building the heap: racking 30 petabytes of hard drives for pretraining
271–280 of 281 posts
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#272Nice writeup. All of the technical detail is great! I'm curious about the process of getting colo space. Did you use a broker? Did you negotiate, and if so, how large was the difference in price between what you initially were quoted and what you ended up paying?
We reached out to almost every colocation space in SF/some in Fremont to get quotes. There wasn't a difference between the quote price and what we ended up paying, though we did negotiate terms + one-time costs.
Please consider posting the quotes, even if you have to redact colo names.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#273Earlier quoted context omitted.
I meant a “single mode, non terminated fiber optic cable from point to point”. In other words, your own cable without any other traffic on it. A shared one will be metro Ethernet in my parlance.
Noob question, is there an in-between where people sell spectrum bands on a shared fiber without going up to L2? Or is that just too complex compared to using off the shelf ethernet and sharing that.
Yep, absolutely, often called "wave/wavelength" or "lambda" service, or "lit fiber".
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#274Earlier quoted context omitted.
Massive props for getting it done anyway. For others reading: In general a switch should never run DHCPd, but will normally/often relay it for you, your arista's would 100% have supported relaying, but in this case it sounds like it might even be flat L2. Normally you'd host dhcpd on a server. Some general feedback incase it's helpful.. -20K on contractors seems insane if we're talking about rack and stack for 10 rac…
def agree on the setup fees, that was just a price crunch to get it done within the weekend. (too short-notice for professional services, too sensitive for craigslist, so basically just paying a bunch of folks we already knew and trusted) for IPXE do you have any reference material you'd recommend? we had 3 people each with reasonably substantial server experience try for like 6 hours each and for whatever reason it…
Honestly, with 10 servers, a pxe setup is probably overkill. If you're getting used servers (and maybe even if not), you might need to poke them with a KVM to set the boot options so that PXE is an option, and you might want to configure the BMC/IPMI from the console too, and then configure anything for serial over IPMI / bios console on serial ports... do that in your office, since your colo is across the street, and then you may as well do the install too. Then when you install, it should just work and crash cart if not. But, PXE is fun, so...
For PXE / iPXE, there's several stages of boot. You have your NIC's option rom, which might be, but probably is not iPXE. That will hit DHCP to get its own IP and also request info about where to pull boot files. You'll need to give it a tftp server IP and a filename. DHCPD config below
I server iPXE executables to non-iPXE. When iPXE starts up, it again asks DHCP, but now you can give it an http boot script. The simplest thing is to have something like
kernel installer_kernel
initrd installer_initrd
boot
Then you'll have your installer (or whatever) booted, and you might have an unattended install setup for that, or you can just setup a rescue image that does dhcp (again!) and opens sshd so you can shell in and do whatever. Up to you.
the pxe part of my isc dhcpd config is:
next-server 203.0.113.11;
if exists user-class and option user-class = "iPXE" {
option ipxe.no-pxedhcp 1;
filename "http://203.0.113.11/tftpboot/menu.ipxe";
} else {
if option client-arch = 00:06 {
filename "ipxe.efi-i386";
} else if option client-arch = 00:07 {
filename "ipxe.efi-x86_64";
} else {
filename "undionly.kpxe";
}
}
And I have those three files in the root of my tftp server. There's all sorts of other stuff you could do, but this should get you started. You don't really need iPXE either, but it's a lot more flexible if you need anything more, and it can load from http which is gobs faster if you have large payloads.
If you really wanted to be highly automated, your image could be fully automated, pull in config from some system and reconfigure the BMC while it was there. But there's no need for that unless you've got tons of servers. Might be something to consider if you mass replace your disk shelves with 4U disk servers, although it might not save a ton of time. If you're super fancy, your colo network would have different vlans and one of them would be the pxe setup vlan --- new servers/servers needed reimaging could be put into the pxe vlan and the setup script could move them into the prod vlan when they're done. That's fun work, but not really needed, IMHO. Semi-automated setup scales a lot farther than people realize, couple hundred servers at least. autopw [2] can help a lot!
[1] https://ipxe.org/howto/dhcpd
[2] https://github.com/jschauma/sshscan/blob/master/src/autopw
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#275For a workload of that size you would be able to negotiate private pricing with AWS or any cloud provider, not just CloudFlare. You can get a private pricing deal on S3 with as little as half a PB. Not saying that your overall expenses would be cheaper w/a CSP than DIY, but its not exactly an apples to apples comparison of taking full retail prices for the CSPs against eBayed equipment and free labor (minus the cost…
What sort of deal are you taking about? Would it be 50% or more?
Not for that minimum 0.5PB volume.
Even at 10PB, the storage commit discounts won't be anywhere near 50%. Probably more like 10-20%, if that.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#276But where do you get 90 million hours worth of video data?
Assuming my calculation is accurate, 90,000,000 hours of video using around 30 PB comes to an average bit rate of about 760k. (Hard to guess though bc I doubt they’re using up all the space they provision day1)
So my guess is either CCTV type of footage where there’s large gaps of motion / high GOP / big codec gains - or something like desktop recordings which are generally very low bit rate even though they can be high res. At that bitrate I can’t imagine it’s something like YouTube video. (Unrelated to the bitrate maybe it’s something like all older public domain videos). I would love to have an idea of what type of videos they are using (just out of curiosity)
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#277Aren't those netapp shelves pretty old at this point? See a lot of people recommending against them even for homelab type uses. You can get those 60 drive SuperMicro JBODs for pretty cheap now, and those aren't too old, would have been my choice. Plus, the TCO is already way under the cloud equiv. so might as well spend a little more to get something much newer and more reliable
yeah it's on the wishlist to try
Thanks to op for actually replying to the various comments here - really appreciate that (and for the initial of course!)
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#278Well done! I love the honest write up and the “can do” attitude. Must have been a lot of fun too. Out of interest why do you think you made the mistake of buying 20x more drives than you needed instead of the denser storage that you mention? Was there a reason you opted for this?
I think <2x more drives than needed, not 20x (24 vs 14TB), but the racks holding the drives could've been denser. Around the same cost in any case and our colo doesn't charge for space, so it's not a big deal and we were just going with what we were familiar with, but something to try.
Oops sorry, my bad! Great to read all about it - good luck with the project.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#279Any startup that has enough money to casually buy a two-letter domain name has too much money, period. Kind of like counting the number of Aeron chairs at startups of old. Not a good sign.
it's .inc... those aren't expensive
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#280You could get pretty close to the cost 1$/TB/month using Hetzner's sx135 with 8x22TB so 140TB in raidz1 for 240 eur. Maybe you get a better rate if you rent 200 of them. Someone else takes care of a lot of risks and you can sleep well at night
Your math does not math. It is more like $2/TB/month with minimal redundancy.