Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

241–250 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#241

Post by jnsaff2 »

jmakov wrote: post #239

Wonder why everybody's first pick is CEPH which is known for being hard to optimize vs e.g. SeaweedFS

If I'd have to guess then I would think that Ceph is the only one who is truly open source and does not feature gate important parts to paid enterprise users.

I did go through this couple of years ago and we ended up with Ceph as well. Combine this with reusing existing hardware that was very suboptimal for Ceph in several ways, it was a pretty bad experience and in the end for our use case AWS was able to offer a good enough pricing that the performance and reliability of S3 was a better deal than managing it ourselves.

If I would do it again then I would make sure that I have the hardware setup that is ideal (plenty of SSD's for metadata, every spinning disk directly addressed as a single OSD, sound network topology and fast enough NIC's) and probably use Rook instead of cephadm. The monitoring, configuration and documentation side of Ceph is however still quite sad, it was really hard to figure out why something is slow and how to tune things faster.

That said, if the Enterprise options are performing better or you at least get good support for tuning and optimizing then the alternatives could be well worth consideration.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#242

Post by Barbing »

azinman2 wrote: post #217

Where does one acquire 90M hours of video without being YouTube?

Anywhere as long as you can avoid “legal/practice/business slog”. Success is defined by $1.5b settlements.

:) just kidding but also curious where besides torrents

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#243

Post by Barbing »

twoodfin wrote: post #188
drnick1 wrote: post #105

Everyone should give AWS the middle finger and start doing this. Beyond cost, it's a matter of sovereignty over one's computing and data.

If this is a real market, I’d expect AWS to introduce S3 Junkyard with a similar durability and cost structure. They probably still won’t budge on the egress fees.

>S3 Junkyard

There it is, the answer to how to mitigate brand damage when risking distance between themselves and some of those 9s.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#244

Post by Barbing »

_1tem wrote: post #18
not--felix wrote: post #14

But where do you get 90 million hours worth of video data?

And not just any video data, they specifically mentioned screen recordings for agentic computer uses. A very specific kind of video. My guess is they have a partnership with someone like Rewind.ai

“For your privacy, your screen and audio recordings are stored locally and NEVER leave your Mac.”

Tell me it’s only someone _like_ Rewind and not actually them! Quoting from the Privacy page they link in their header.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#245

Post by dist-epoch »

antisthenes wrote: post #178
joering2 wrote: post #92

Earlier quoted context omitted.

Their rate will probably be higher since they are utilizing used drives. From the spec: 2,400 drives. Mostly 12TB used enterprise drives (3/4 SATA, 1/4 SAS). The JBOD DS4246s work for either.

Not necessarily, since disk failures are typically U-shaped. Buying used drives eliminates the high rate of early failure (but does get you a bit closer to the 2nd part of the U-curve). Typically most drives would become more obsolete before hitting the high failure rate of the right side of the U-curve from longevity (7+ years)

Physically moving the drive tends to reset the U-shape. Some will be damaged.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#246

Post by woooooo »

bayindirh wrote: post #125
dpe82 wrote: post #116

Earlier quoted context omitted.

I think these days folks say "dark fiber" for any kind of connection you buy. It bothers me too.

I meant a “single mode, non terminated fiber optic cable from point to point”. In other words, your own cable without any other traffic on it. A shared one will be metro Ethernet in my parlance.

Noob question, is there an in-between where people sell spectrum bands on a shared fiber without going up to L2? Or is that just too complex compared to using off the shelf ethernet and sharing that.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#248

Post by hengheng »

Barbing wrote: post #244
_1tem wrote: post #18

Earlier quoted context omitted.

And not just any video data, they specifically mentioned screen recordings for agentic computer uses. A very specific kind of video. My guess is they have a partnership with someone like Rewind.ai

“For your privacy, your screen and audio recordings are stored locally and NEVER leave your Mac.” Tell me it’s only someone _like_ Rewind and not actually them! Quoting from the Privacy page they link in their header.

> we prob don't want to have it in Europe

Best indicator as to what that is.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#249

Post by rtp4me »

g413n wrote: post #199
matt-p wrote: post #187

Earlier quoted context omitted.

Massive props for getting it done anyway. For others reading: In general a switch should never run DHCPd, but will normally/often relay it for you, your arista's would 100% have supported relaying, but in this case it sounds like it might even be flat L2. Normally you'd host dhcpd on a server. Some general feedback incase it's helpful.. -20K on contractors seems insane if we're talking about rack and stack for 10 rac…

def agree on the setup fees, that was just a price crunch to get it done within the weekend. (too short-notice for professional services, too sensitive for craigslist, so basically just paying a bunch of folks we already knew and trusted) for IPXE do you have any reference material you'd recommend? we had 3 people each with reasonably substantial server experience try for like 6 hours each and for whatever reason it…

I have done a ton of iPXE boot setups in the past. We use iPXE at our DC location for imaging, system recovery, etc. In fact, I just finished up a new boot image that creates a 100MB virtual floppy drive used for BIOS updates. Reach out and I can provide the entire setup if you like (pxe config files, boot loaders, scripts, etc).

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#250

Post by rtp4me »

Aurornis wrote: post #65
ttfvjktesd wrote: post #35

The biggest part that is always missing in such comparisons is the employee salaries. In the calculation they give $354k/year of total cost per year. But now add the cost of staff in SF to operate that thing.

Small startup teams can sometimes get away with datacenter management being a side task that gets done on an as-needed basis at first. It will come with downtime and your stability won't be anywhere near as good as Cloudflare or AWS no matter how well you plan, though. Every real-world colocation or self-hosting project I've ever been around has underestimate their downtime and rate of problems by at least an order o…

For drive issues, this is easy. Have a stack of replacements on hand and just open a "remote-hands" ticket with the CoLo provider to swap out the drive. This can usually be done in 1-2hrs from opening the ticket.

For server issues; again, pretty easy. Just use iKVM/IPMI and iPXE to diagnose a faulty server. Again, using "remote-hands" from the CoLo provider can help fix problems if your staff does not have the skills.

Post reply on HN