Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

121–130 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#121

Post by dpe82 »

stackskipton wrote: post #30
mschuster91 wrote: post #20

Shows how crazy cheap on prem can be. tips hat

Not included is overhead of dealing with maintenance. S3/R2 generally don’t require OPS type dedicated to care and feeding. This type of setup will likely require someone to spend 5 hours a week dealing with it.

a) 5hrs/week is negligible compared to that potential AWS bill.

b) The seem tolerant of failures so it's not going to be anything like 5hrs/week of physical maintenance. It will be bursty though (eg. box died, time to replace it...) but assuming they have spares of everything sitting around / already racked it shouldn't be a big deal.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#122

Post by p_ing »

antonkochubey wrote: post #42
p_ing wrote: post #29

Earlier quoted context omitted.

You're in a seismically active part of the world. Will the venture last in a total loss scenario?

They spent $300,000 on drives, with AWS they would have spent 4x that PER MONTH. They're already ahead of the cloud.

AWS/cloud doesn't factor into my question what so ever. Loss of equipment is one thing. Loss of all data is quite a different story.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#123

Post by tempest_ »

pronoiac wrote: post #78

I wonder if they'll go with "toploaders" - like Backblaze Storage Pods - later. They have better density and faster setup, as they don't have to screw in every drive. They got used drives. I wonder if they did any testing? I've gotten used drives that were DOA, which showed up in tests - SMART tests, short and long, then writing pseudorandom data to verify capacity.

Used Supermicro machines of this generation and very cheap (all things considered)

https://www.theserverstore.com/supermicro-superstorage-ssg-6...

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#124

Post by p_ing »

nee1r wrote: post #33
p_ing wrote: post #29

Earlier quoted context omitted.

You're in a seismically active part of the world. Will the venture last in a total loss scenario?

We're currently 1/1 for the recent 4.3 magnitude earthquake (though if SF crumbles we might lose data)

4.3 is a baby quake. I'd hope that you'd be 1/1!

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#125

Post by bayindirh »

dpe82 wrote: post #116
geor9e wrote: post #74

Earlier quoted context omitted.

Does San Francisco really still have dark fiber? That 90s bubble sure did overshoot demand.

I think these days folks say "dark fiber" for any kind of connection you buy. It bothers me too.

I meant a “single mode, non terminated fiber optic cable from point to point”. In other words, your own cable without any other traffic on it.

A shared one will be metro Ethernet in my parlance.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#126

Post by bombcar »

buzer wrote: post #111
trebligdivad wrote: post #97

The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.' Where doe…

> Wth would you want public IPs. Possibly to avoid needing NAT (or VPN) gateway that can handle 100Gbps.

I don't know what they're doing, but Mikrotik can perhaps route that → https://mikrotik.com/product/ccr2216_1g_12xs_2xq#fndtn-testr... and is about the cost of their used thing.

And I think this would be a banger for IPv6 if they really "need" public IPs.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#127

Post by matt-p »

trebligdivad wrote: post #97

The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.' Where doe…

This was not written by a network person, quite clearly. Hopefully it's just a misunderstanding, otherwise they do need someone with literally any clue about networks.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#128

Post by nodja »

yread wrote: post #119

You could get pretty close to the cost 1$/TB/month using Hetzner's sx135 with 8x22TB so 140TB in raidz1 for 240 eur. Maybe you get a better rate if you rent 200 of them. Someone else takes care of a lot of risks and you can sleep well at night

I don't think Hetzner provides locations in SF. Those 100GBit connections don't do much if they need to connect outside the city the rest of the equipment is in, but maybe peering has gotten better and my views are outdated.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#129

Post by dustywusty »

bombcar wrote: post #126
buzer wrote: post #111

Earlier quoted context omitted.

> Wth would you want public IPs. Possibly to avoid needing NAT (or VPN) gateway that can handle 100Gbps.

I don't know what they're doing, but Mikrotik can perhaps route that → https://mikrotik.com/product/ccr2216_1g_12xs_2xq#fndtn-testr... and is about the cost of their used thing. And I think this would be a banger for IPv6 if they really "need" public IPs.

Exactly what I came in to say, CCR2216 can do this for < $2k, and does it well.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#130

Post by g413n »

yread wrote: post #119

You could get pretty close to the cost 1$/TB/month using Hetzner's sx135 with 8x22TB so 140TB in raidz1 for 240 eur. Maybe you get a better rate if you rent 200 of them. Someone else takes care of a lot of risks and you can sleep well at night

yeah it's totally plausible that we go with something like this in the future. We have similar offers where we could separate out either the financing, the build-out, or both and just do the software.

(for Hetzner in particular it was a massive pain when we were trying to get CPU quotas with them for other data operations, and we prob don't want to have it in Europe, but it's been pretty easy to negotiate good quotes on similar deals locally now that we've shown we can do it ourselves)

Post reply on HN