Colossus for Rapid Storage • Hacker News Board

Re: Colossus for Rapid Storage

#31

Post by decimalenough » Thu, Apr 10, 2025, 4:59 AM UTC

simonw wrote: post #23

Everyone needs to learn to use a single, unique, unambiguous URL for new product announcements like this. Google aren't the only company that consistently mess this up, but given how they built a 1.95 trillion company on top of crawling URLs on the web they really should have an internal culture that values giving things unique URLs! [I had to learn this lesson myself: I used to blog "weeknotes" every week or two whe…

Google's not really at fault here: the OP submitted a link to an article called "Introducing Ironwood TPUs and new innovations in AI Hypercomputer" that happens to mention Rapid Storage way down the page.

In case marketing seems to move faster than documentation though, since I can't find any mention of this in the main GCS docs. https://cloud.google.com/search?hl=en&q=rapid%20storage

Re: Colossus for Rapid Storage

#32

Post by korkybuchek » Thu, Apr 10, 2025, 5:02 AM UTC

jeffbee wrote: post #29

alobrah wrote: post #4

FYI this was unveiled at the 2025 Google Next conference, and they're apparently unveiling a gRPC client for Rapid Storage, which appears to be a very thin wrapper over Colossus itself, as this is just zonal storage.

What on this page gives you that impression? Do I have to watch the 2-hour video to learn this?

Of course not. Gemini can summarize it for you.

Re: Colossus for Rapid Storage

#33

Post by wetpaws » Thu, Apr 10, 2025, 5:06 AM UTC

[dead]

Re: Colossus for Rapid Storage

#34

Post by nodesocket » Thu, Apr 10, 2025, 5:15 AM UTC

acstorage wrote: post #9

Similar to S3 express one zone

Is S3 Express One Zone performance greatly improved to standard S3 like GCP rapid storage? My understanding is S3 Express One Zone is just more cost effective.

> 20x faster random-read data loading than a Cloud Storage regional bucket.

Re: Colossus for Rapid Storage

#35

Post by CobrastanJorji » Thu, Apr 10, 2025, 5:17 AM UTC

korkybuchek wrote: post #32

jeffbee wrote: post #29

Earlier quoted context omitted.

What on this page gives you that impression? Do I have to watch the 2-hour video to learn this?

Of course not. Gemini can summarize it for you.

I mean, sure, it can easily provide quick text summaries of this sort of thing, but I only consume ML summaries in the forms of podcast discussions between two simulated pundits, as God intended.

Re: Colossus for Rapid Storage

#36

Post by Foobar8568 » Thu, Apr 10, 2025, 5:18 AM UTC

klabb3 wrote: post #13

hiddencost wrote: post #7

Hats off to whoever convinced management that selling Colossus via cloud was Artificial Intelligence. Bravo.

That’s an actually impressive level of spin. Flashback to when shipping companies were slapping blockchain on international container shipments.

Flashback to the cabbage on the Blockchain or when they wanted to tag each fish caught in the wild as well for hu traceability!

Re: Colossus for Rapid Storage

#37

Post by nodesocket » Thu, Apr 10, 2025, 5:19 AM UTC

nodesocket wrote: post #34

acstorage wrote: post #9

Similar to S3 express one zone

Is S3 Express One Zone performance greatly improved to standard S3 like GCP rapid storage? My understanding is S3 Express One Zone is just more cost effective. > 20x faster random-read data loading than a Cloud Storage regional bucket.

Update: Just read this article[1] which clarifies S3 Express One Zone. Yes, performance is greatly improved, but actually storage costs are 8x more than a standard S3 bucket. The naming S3 Express One Zone is terrible and a bit misleading on pricing changes.

[1] https://www.warpstream.com/blog/s3-express-is-all-you-need

Re: Colossus for Rapid Storage

#38

Post by jashmatthews » Thu, Apr 10, 2025, 5:19 AM UTC

nodesocket wrote: post #34

acstorage wrote: post #9

Similar to S3 express one zone

Is S3 Express One Zone performance greatly improved to standard S3 like GCP rapid storage? My understanding is S3 Express One Zone is just more cost effective. > 20x faster random-read data loading than a Cloud Storage regional bucket.

AWS claims 10x lower latency but I haven't personally checked.

Re: Colossus for Rapid Storage

#39

Post by dangoodmanUT » Thu, Apr 10, 2025, 5:20 AM UTC

nodesocket wrote: post #34

acstorage wrote: post #9

Similar to S3 express one zone

Is S3 Express One Zone performance greatly improved to standard S3 like GCP rapid storage? My understanding is S3 Express One Zone is just more cost effective. > 20x faster random-read data loading than a Cloud Storage regional bucket.

Yes, it’s horribly more expensive… I think you are thinking of one zone infrequent access

Re: Colossus for Rapid Storage

#40

Post by CobrastanJorji » Thu, Apr 10, 2025, 5:21 AM UTC

hiddencost wrote: post #7

Hats off to whoever convinced management that selling Colossus via cloud was Artificial Intelligence. Bravo.

They're not saying that it's AI. They're saying it's for customers who do AI. Training means lots and lots of reads from a big data store, and if you're reading from, like, big Parquet files, that probably means lots of random reads. This is for that. Speedier data access, presumably at the cost of durability and availability, which is probably a great trade-off for people doing ML training jobs.