191–200 of 281 posts
Earlier quoted context omitted.
Isn't the easy option to spread the computers out, i.e. not fill the rack, but only half of it? A GPU cluster next to my servers has done this, presumably they couldn't have 64A in one rack so they've got 32A in two. (230V 3phase.)
Rackspace is typically at a premium at most data centers.
Earlier quoted context omitted.
Docker is amazing for forcing the machines not to be pets, seriously, a racked sever is just another K3 or K8 node (or whatever) and doesn't get the choice or ability of being petted. It's so nice. You could maybe of said the same about vm's but not really, the VM just became the pet, OK you could at least image/snapshot it but it's not the same.
I've found docker is as of a monstrous pet. Docker is a monster that you have to treat as a pet. You've still got to pet it through stages of updating, monitoring, snapshots and networking. When the internal system breaks it's no different to a server collapsing. Snapshots are a haircut for the monster, useful but can make things worse.
Earlier quoted context omitted.
How did you arrive at the decision of not putting the GPU machines in the colo? Were the power costs going to be too high? Or do you just expect to need more physical access to the GPU machines vs the storage ones?
When I was working at sfcompute prior to this we saw multiple datacenters literally catch on fire bc the industry was not experienced with the power density of h100s. Our training chips just aren't a standard package in the way JBODs are.
Where is that done? How many GPUs do you need to crunching all that data. Etc.
Very interesting and refreshing read though. Feels like what Silicon Valley is more about than just the usual: tf apply then smile and dial.
Earlier quoted context omitted.
If I remember correctly, most drives either: 1. Fail in the first X amount of time 2. Fail towards the end of their rated lifespan So buying used drives doesn't seem like the worst idea to me. You've already filtered out the drivers that would fail early. Disclaimer: I have no idea what I'm talking about
we don't have perfect metrics here but this seems to match our experience; a lot of failures happened shortly after install before the bulk of the data download onto the heap, so actual data loss is lower than hardware failure rates
Earlier quoted context omitted.
Yeah from memory on-prem was always cheaper, it just removed a lot of logistic obstacles and made everything convenient under one bill. IIRC the wisdom of the time cloud started becoming popular was to always be on-prem and use cloud to scale up when demand spiked. But over time temporarily scaling up became permanent, and devs became reliant on instantly spawning new machines for things other than spikes in demand a…
> IIRC the wisdom of the time cloud started becoming popular was to always be on-prem and use cloud to scale up when demand spiked. I've heard that before but was never able to make sense of it. Overflowing into the cloud seems like a nightmare to manage, wouldn't overbuilding on-prem be cheaper than paying your infra team to straddle two environments?
The end state is “just some IaC,” wherein it doesn’t really matter to anyone where the application lives, but all of the underlying difficulties in getting to that state necessitate that your team actually, no-shit knows how distributed systems work. They’re going to be doing a lot of networking configuration, for one, and that’s a whole speciality.
Fun piece, thanks to the author. But for vicarious thrills like this, more pictures are always appreciated!
If the authors chime in, I'd like to ask what "Standard Intelligence PBC" does. Is it a public benefit corp? What are y'all building?
We're working on pretraining computer action models from the ground up—hence the pretraining data cluster. We're a public benefit corp because we think its important for AGI to built in the public's interest + are planning on automating a lot of the work done on computers!
Earlier quoted context omitted.
If the authors chime in, I'd like to ask what "Standard Intelligence PBC" does. Is it a public benefit corp? What are y'all building?
We did want more pictures!! Recently bought a Sony A7III to capture more fun moments like this. We're working on pretraining computer action models from the ground up—hence the pretraining data cluster. We're a public benefit corp because we think its important for AGI to built in the public's interest + are planning on automating a lot of the work done on computers!
Earlier quoted context omitted.
I've found docker is as of a monstrous pet. Docker is a monster that you have to treat as a pet. You've still got to pet it through stages of updating, monitoring, snapshots and networking. When the internal system breaks it's no different to a server collapsing. Snapshots are a haircut for the monster, useful but can make things worse.
Not in my experience, super easy to setup a K3s cluster in a single rack. Certainly less hassle than VMWare was or XEN ever was.
Some can tame the beast, for me it's always a fight until one of us holds down the power button.
Earlier quoted context omitted.
yeah misunderstanding we'll update the post-- separately it's true that we aren't network specialists and the network wrangling was prob disproportionately hard for us/ shouldn't have taken so long.
Massive props for getting it done anyway. For others reading: In general a switch should never run DHCPd, but will normally/often relay it for you, your arista's would 100% have supported relaying, but in this case it sounds like it might even be flat L2. Normally you'd host dhcpd on a server. Some general feedback incase it's helpful.. -20K on contractors seems insane if we're talking about rack and stack for 10 rac…
for IPXE do you have any reference material you'd recommend? we had 3 people each with reasonably substantial server experience try for like 6 hours each and for whatever reason it turned out to be too difficult.
Earlier quoted context omitted.
> Our reliability came from software not hardware, though of course we had hundreds of spares sitting by, the defense in depth (multiple datacenters, each datacenter having 2 'brains' which could hotswap, each client multiply backed up on 3-4 machines)... Of course, but building and managing the software stack, managing hundreds of spares across locations, spanning across datacenters, having a hotswap backup system i…
To be fair, what's described here is much more robust than what you get with a simple AWS setup. At a minimum that's a multi-region setup, but if the DCs have different owners I'd even compare it to a multi-cloud setup.