101–110 of 281 posts
i am still confused what their software stack is, they dont use ceph but bought netapp, so they use nfs?
The NetApps are just disk shelves, can plug it into a SAS controller and use whatever software stack you please.
Earlier quoted context omitted.
How about all the other infrastructure. Since you are obviously not using the cloud, you must have massive amounts of GPUs and operating systems. All of that has been working together, it's not just keep watching for the physical disks and all is set. Don't get me wrong, I buy the actual numbers regarding hardware costs, but in addition to that presenting the rest as basically a one man show in terms of maintenance h…
oh we use cloud gpus, infiniband h100s absolutely aren't something we want to self-host. not aws tho, they're crazy overpriced; mithril and sfcompute! we also use cloudflare extensively for everything that isn't the core heap dataset, the convenience of buckets is totally worth it for most day-to-day usage. the heap is really just the main pretraining corpus and nothing else.
My question isn't why do it yourself. A quick back of the envelope math shows AWS being much more expensive. My question is why San Francisco? It's one of the most expensive real estate markets in the US (#2 residential, #1 commercial), and electricity is expensive . $0.71/KwH peak residential rate! A jaunt down 280 to San Jose's gonna be cheaper, at the expense of. having to take that drive to get hands on. But I'm…
Earlier quoted context omitted.
> Our reliability came from software not hardware, though of course we had hundreds of spares sitting by, the defense in depth (multiple datacenters, each datacenter having 2 'brains' which could hotswap, each client multiply backed up on 3-4 machines)... Of course, but building and managing the software stack, managing hundreds of spares across locations, spanning across datacenters, having a hotswap backup system i…
not caring about redundancy/reliability is really nice, each healthy HDD is just the same +20TB of pretraining data and every drive lost is the same marginal cost.
Earlier quoted context omitted.
The NetApps are just disk shelves, can plug it into a SAS controller and use whatever software stack you please.
but they have multiple head nodes, so its some distributed setup or just active/passive type thing?
No mention of disk failure rates? curious how it's holding up after a few months
I've mentioned this story before, but we had massive drive failures when bringing up multiple disk arrays. We get them racked on a friday afternoon, and then I wrote a quick and dirty shell script to read/write data back and forth between them over the weekend that was to kick in after they finished striping the raid arrays. By quick and dirty I mean there was no logging, and just a bunch of commands saved as .sh. Ca…
Heh, that's a clever solution to the problem of managing storage through the full 10 year disk lifecycle.
Earlier quoted context omitted.
They mentioned the cluster being used enterprise drives, I can see the desire to save money but agree, that is going to be one expensive mistake down the road. I should also note personally for home cluster use, I learned quickly that used drives didn’t seem to make sense. Too much performance variability.
If I remember correctly, most drives either: 1. Fail in the first X amount of time 2. Fail towards the end of their rated lifespan So buying used drives doesn't seem like the worst idea to me. You've already filtered out the drivers that would fail early. Disclaimer: I have no idea what I'm talking about
Still fun for someone wanting to stick a computer in a DC though.
Networking is surprisingly hard but we also settled for the cheapo life QSFP instead of the new Cisco switches that do 800 Gbps that are coming. Great writeup.
One that would be fun is about the mechanics of layout and cabling and that sort of thing. Learning all that manually was a pain in the ass. It's not just written down somewhere and I should have done it when I was doing it but now I no longer am doing it and so can't provide good photos.