Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

111–120 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#111

Post by buzer »

trebligdivad wrote: post #97

The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.' Where doe…

> Wth would you want public IPs.

Possibly to avoid needing NAT (or VPN) gateway that can handle 100Gbps.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#112

Post by ThinkBeat »

So now you have all

- your storage in one place

- you own all backup,

-- off site backup (hot or cold)

- uptime worries

- maintenance drives

-- how many can fail. before it is a problem

- maintenance machines

-- how many can fail. before it is a problem

- maintenance misc/datacenter

- What to do the electricity is cut off suddenly

-- do you have a backup provider?

-- disel generators?

-- giant batteries?

-- Will the backup power also run cooling?

-natural disaster

-- earthquake

-- flooding

-- heatwave

- physical security

- employee training / (esp. if many quit)

- backup for networking (and power for it)

- employees on call 24/7

- protection against hacking

+++++

I agree that a lot of cloud providers overcharge by a lot, but doing it all yourself gives you a lot of headaches.

co-hosting would seem like a valuable partial mitigator.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#113

Post by tempest_ »

theideaofcoffee wrote: post #81
jimmytucson wrote: post #75

Just wanted to say, thanks for doing this! Now the old rant... I started my career when on-prem was the norm and remember so much trouble. When you have long-lived hardware, eventually, no matter how hard you try, you just start to treat it as a pet and state naturally accumulates. Then, as the hardware starts to be not good enough, you need to upgrade. There's an internal team that presents the "commodity" interface…

I'm not op, but thanks for this. Like I mentioned in another comment, the wholesale move to the cloud has caused so many skills to become atrophied. And it's good that someone is starting to exercise that skill again, like you said. The hyperscalers are mostly to blame for this, the marketing FUD being that you can't possibly do it yourself, there are too many things to keep track of, let us do it (while conveniently…

The other thing the cloud does not let you do is make trade offs.

Sometimes you can afford not to have triple redundant 1000GB network or a simple single machine with raid may have acceptable down time.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#114

Post by renewiltord »

fragmede wrote: post #95

My question isn't why do it yourself. A quick back of the envelope math shows AWS being much more expensive. My question is why San Francisco? It's one of the most expensive real estate markets in the US (#2 residential, #1 commercial), and electricity is expensive . $0.71/KwH peak residential rate! A jaunt down 280 to San Jose's gonna be cheaper, at the expense of. having to take that drive to get hands on. But I'm…

Problem when you self-roll this is that you inevitably make mistakes and the cycle time of going down and up ruins everything. Access trumps everything.

You can get a DC guy but then he doesn't have much to do post setup and if you contract that you're paying mondo dollars anyway to get it right and it's a market for lemons (lots of bullshitters out there who don't know anything).

Learned this lesson painfully.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#115

Post by pclmulqdq »

ThinkBeat wrote: post #112

So now you have all - your storage in one place - you own all backup, -- off site backup (hot or cold) - uptime worries - maintenance drives -- how many can fail. before it is a problem - maintenance machines -- how many can fail. before it is a problem - maintenance misc/datacenter - What to do the electricity is cut off suddenly -- do you have a backup provider? -- disel generators? -- giant batteries? -- Will the…

Most of these come from your colo provider (including a good backup power and networking story), and you can pay remote hands for a lot of the rest.

Things like "protection from hacking" also don't come from AWS.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#116

Post by dpe82 »

geor9e wrote: post #74
bayindirh wrote: post #7

Earlier quoted context omitted.

They can rent a dark fiber for themselves for that distance, and it'll be cheap. However, as they noted they use 100gbps capacity from their ISP.

Does San Francisco really still have dark fiber? That 90s bubble sure did overshoot demand.

I think these days folks say "dark fiber" for any kind of connection you buy. It bothers me too.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#117

Post by pclmulqdq »

trebligdivad wrote: post #97

The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.' Where doe…

They didn't seem to want to use a router. Purpose-built 100 Gbps routers are a bit expensive, but you can also turn a computer into one.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#118

Post by mystifyingpoi »

trebligdivad wrote: post #97

The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.' Where doe…

It really feels like they wanted 30 PB of storage accessible over HTTP and literally nothing else. No redundancy, no NAT, dead simple nginx config + some code to track where to find which file on the filesystem. I like that.
Post reply on HN