Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

201–210 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#201

Post by jandrewrogers »

luhn wrote: post #174
nodja wrote: post #133

Earlier quoted context omitted.

Yeah from memory on-prem was always cheaper, it just removed a lot of logistic obstacles and made everything convenient under one bill. IIRC the wisdom of the time cloud started becoming popular was to always be on-prem and use cloud to scale up when demand spiked. But over time temporarily scaling up became permanent, and devs became reliant on instantly spawning new machines for things other than spikes in demand a…

> IIRC the wisdom of the time cloud started becoming popular was to always be on-prem and use cloud to scale up when demand spiked. I've heard that before but was never able to make sense of it. Overflowing into the cloud seems like a nightmare to manage, wouldn't overbuilding on-prem be cheaper than paying your infra team to straddle two environments?

I've setup on-prem environments that would automatically and transparently spill workload to the cloud when we ran out of on-prem capacity. The prerequisite is that your operations people need to have strong automation skills, there can't be any human button-pushers in the loop, but it really isn't that much work nor does it require unusual technical skill if you have a competent operations environment. Once everything is wired up, it mostly just works. Even the devs often don't know whether their workloads are running on-prem or in the cloud.

An under-rated aspect of this is that it provides detailed data for planning the expansion of the on-prem environment, including changes in how the hardware is being used.

My experience is that this is a good model. On-prem is about 1/3 the cost of the cloud, fully burdened, in my experience but being able to transparently spill to the cloud mitigates the capacity and supply chain risks that come with on-prem.

It is effectively a cheap insurance policy.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#202

Post by xp84 »

buzer wrote: post #143
xp84 wrote: post #139

Earlier quoted context omitted.

No DHCP doesn't mean public IPs nor impact the need for NAT, it just means the hosts have to be explicitly configured with IP addresses, default gateways if they need egress, and DNS. Those IPs you end up assigning manually could be private ones or routable ones. If private, authorized traffic could be bridged onto the network by anything, such as a random computer with 2 NICs, one of which is connected eventually to…

I know, I was specifically answering the question of "why the hell would you want public IPs". I don't know why their network setup wouldn't support DHCP, that's extremely common especially in "enterprise" switches via DHCP forwarding.

Ok then yes I agree with you. That was weird

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#204

Post by randomfinn »

g413n wrote: post #87
Sanzig wrote: post #62

Earlier quoted context omitted.

Did you do any kind of redundancy at least (eg: putting every 10 disks in RAID 5 or RAID Z1)? Or I suppose your training application doesn't mind if you shed a few terabytes of data every so often?

atm we don't and we're a bit unsure whether it's a free lunch wrt adding complexity. there's a really nice property of having isolated hard drives where you can take any individual one and `sudo mount` it and you have a nice chunk of training data, and that's something anyone can feel comfortable touching without any onboarding to some software stack

I wonder if snapraid would work for this. Especially if your data is mostly written once and then just read, it could be an easy way to add redundancy while keeping isolated individual drives.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#205

Post by epistasis »

coleca wrote: post #131

For a workload of that size you would be able to negotiate private pricing with AWS or any cloud provider, not just CloudFlare. You can get a private pricing deal on S3 with as little as half a PB. Not saying that your overall expenses would be cheaper w/a CSP than DIY, but its not exactly an apples to apples comparison of taking full retail prices for the CSPs against eBayed equipment and free labor (minus the cost…

What sort of deal are you taking about? Would it be 50% or more?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#207

Post by nee1r »

tarasglek wrote: post #101
OliverGuy wrote: post #99

Earlier quoted context omitted.

The NetApps are just disk shelves, can plug it into a SAS controller and use whatever software stack you please.

but they have multiple head nodes, so its some distributed setup or just active/passive type thing?

We have a custom barebones solution that uses a hashring to route the files!

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#208

Post by iJohnDoe »

matt-p wrote: post #168
jimmytucson wrote: post #75

Just wanted to say, thanks for doing this! Now the old rant... I started my career when on-prem was the norm and remember so much trouble. When you have long-lived hardware, eventually, no matter how hard you try, you just start to treat it as a pet and state naturally accumulates. Then, as the hardware starts to be not good enough, you need to upgrade. There's an internal team that presents the "commodity" interface…

Docker is amazing for forcing the machines not to be pets, seriously, a racked sever is just another K3 or K8 node (or whatever) and doesn't get the choice or ability of being petted. It's so nice. You could maybe of said the same about vm's but not really, the VM just became the pet, OK you could at least image/snapshot it but it's not the same.

It’s interesting everyone having different experiences and those experiences drive what they do.

I would never dream of running Docker in production. It seems so overly complicated. Also, since day one, I could never understand using a public registry for mission critical stuff. When I was learning Docker, I would unplug the network cable so I wouldn’t accidentally push my container online somewhere with all my data.

I totally get the concept at scale. I also get the concept of just shipping an application in a container. I also get the concept of self-hosting of just give me the container so I don’t have to think about how it all works.

However, the complexity of building the container, cleanup, deleting entries, environment variables, no SSH availability, even on Railway in the beginning, ambiguous where your container needs to be to be to even get it somewhere. Public registry or private registry.

Certainly most of it is my lack of knowledge of not sticking with it.

Just give me a VM and some firewall rules. Cloning VMs can be automated in so many different ways.

/rant

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#209

Post by typpilol »

g413n wrote: post #199
matt-p wrote: post #187

Earlier quoted context omitted.

Massive props for getting it done anyway. For others reading: In general a switch should never run DHCPd, but will normally/often relay it for you, your arista's would 100% have supported relaying, but in this case it sounds like it might even be flat L2. Normally you'd host dhcpd on a server. Some general feedback incase it's helpful.. -20K on contractors seems insane if we're talking about rack and stack for 10 rac…

def agree on the setup fees, that was just a price crunch to get it done within the weekend. (too short-notice for professional services, too sensitive for craigslist, so basically just paying a bunch of folks we already knew and trusted) for IPXE do you have any reference material you'd recommend? we had 3 people each with reasonably substantial server experience try for like 6 hours each and for whatever reason it…

I assume it was their first time setting up ipxe? There's a lot of hang nails with it depending on the infra you're using it in.

For 10 racks it might not make sense.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#210

Post by dgfitz »

iJohnDoe wrote: post #208
matt-p wrote: post #168

Earlier quoted context omitted.

Docker is amazing for forcing the machines not to be pets, seriously, a racked sever is just another K3 or K8 node (or whatever) and doesn't get the choice or ability of being petted. It's so nice. You could maybe of said the same about vm's but not really, the VM just became the pet, OK you could at least image/snapshot it but it's not the same.

It’s interesting everyone having different experiences and those experiences drive what they do. I would never dream of running Docker in production. It seems so overly complicated. Also, since day one, I could never understand using a public registry for mission critical stuff. When I was learning Docker, I would unplug the network cable so I wouldn’t accidentally push my container online somewhere with all my data.…

I agree with you, but I also don’t do webdev. Docker seems neat, I’ve dabbled just to understand the stack and how it works. Not much of a container person I realized. Closest I get is spinning out yocto images and flashing them to hardware. Same image every time with updated versions of software. So maybe kind of like docker in the abstract sense, I suppose.

I’m mid-way through my career at this point and I’m hoping to skip the web/cloud bit, I just don’t enjoy it as much as I do hw/sw integrations.

Post reply on HN