“Solve computer use” and previous work is audio conversation model. How do these go together? Is the idea to replace keyboard and mouse with spoken commands? a la Star Trek
Building the heap: racking 30 petabytes of hard drives for pretraining
141–150 of 281 posts
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#142> We kept this obsessively simple instead of using MinIO or Ceph because we didn’t need any of the features they provided; it’s much, much simpler to debug a 200-line program than to debug Ceph, and we weren’t worried about redundancy or sharding. All our drives were formatted with XFS.
What do you plan to do if you start getting corruption and bitrot? The complexity of S3 comes with a lot of hard guarantees for data integrity.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#143Earlier quoted context omitted.
> Wth would you want public IPs. Possibly to avoid needing NAT (or VPN) gateway that can handle 100Gbps.
No DHCP doesn't mean public IPs nor impact the need for NAT, it just means the hosts have to be explicitly configured with IP addresses, default gateways if they need egress, and DNS. Those IPs you end up assigning manually could be private ones or routable ones. If private, authorized traffic could be bridged onto the network by anything, such as a random computer with 2 NICs, one of which is connected eventually to…
I know, I was specifically answering the question of "why the hell would you want public IPs".
I don't know why their network setup wouldn't support DHCP, that's extremely common especially in "enterprise" switches via DHCP forwarding.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#144>We threw a hard drive stacking party in downtown SF and got our friends to come, offering food and custom-engraved hard drives to all who helped. The hard drive stacking started at 6am and continued for 36 hours (with a break to sleep), and by the end of that time we had 30 PB of functioning hardware racked and wired up.
So how many actual man hours for 2400 drives?
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#145>We threw a hard drive stacking party in downtown SF and got our friends to come, offering food and custom-engraved hard drives to all who helped. The hard drive stacking started at 6am and continued for 36 hours (with a break to sleep), and by the end of that time we had 30 PB of functioning hardware racked and wired up. So how many actual man hours for 2400 drives?
around 250
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#146Earlier quoted context omitted.
Already filled up and looking to possibly copy and paste :)
So, others have asked, and I'm curious myself are you sourcing the videos yourselves or third parties?
My guess would be they are running some dummy app like quote of the day or something and it records the screen at 1fps or so.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#147The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.' Where doe…
This was not written by a network person, quite clearly. Hopefully it's just a misunderstanding, otherwise they do need someone with literally any clue about networks.
yeah misunderstanding we'll update the post-- separately it's true that we aren't network specialists and the network wrangling was prob disproportionately hard for us/ shouldn't have taken so long.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#148The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.' Where doe…
They didn't seem to want to use a router. Purpose-built 100 Gbps routers are a bit expensive, but you can also turn a computer into one.
Many switches are L3 capable, making them in effect a router. Considering their internet lines appear to be hooked up to their 100 Gbps switch, I'd guess this is one of the L3 ones.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#149Earlier quoted context omitted.
I'm not op, but thanks for this. Like I mentioned in another comment, the wholesale move to the cloud has caused so many skills to become atrophied. And it's good that someone is starting to exercise that skill again, like you said. The hyperscalers are mostly to blame for this, the marketing FUD being that you can't possibly do it yourself, there are too many things to keep track of, let us do it (while conveniently…
The other thing the cloud does not let you do is make trade offs. Sometimes you can afford not to have triple redundant 1000GB network or a simple single machine with raid may have acceptable down time.
yeah this
it means that even after negotiating much better terms than baseline we run into the fact that cloud providers just have a higher cost basis for the more premium/general product.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#150Cool write-up.
I do feel sorry for the friends that go suckered into doing a bunch of grunt work for free though