(This was posted last night with https://cloud.google.com/blog/products/compute/whats-new-wit... above. We've changed the URL to the product-specific article.)
Colossus for Rapid Storage
91–100 of 129 posts
Re: Colossus for Rapid Storage
#92Very cool! This makes Google the only major cloud that has low-latency single-zone object storage, standard regional object storage, and transparently-replicated dual-region object storage - all with the same API. For infra systems, this is great: code against the GCS API, and let the user choose the cost/latency/durability tradeoffs that make sense for their use case.
> This makes Google the only major cloud that has low-latency single-zone object storage, standard regional object storage,
Absurd claim. S3 Express launched last year.
Re: Colossus for Rapid Storage
#93Re: Colossus for Rapid Storage
#94Earlier quoted context omitted.
If Apple's so great at naming things, tell me (without looking) which is bigger/better/faster for their CPUs: Max or Ultra?
ha, ha, fair. Ultra. To be fair, I own a MacBook Pro M1 Max and Mac Mini M4 Pro and follow Apple products closely.
Yep, I love Apple, follow them closely, own a Mac Studio with an M3 Ultra and a MacBook Pro with an M4 Max, and it's still confusing. :)
I mean, surely a Mac Studio with an M4 Max must be the best, right? It's an entire CPU generation ahead and it's maximum! Of course, it's not... the M3 Ultra is the best.
Naming things is hard.
Re: Colossus for Rapid Storage
#95Very cool! This makes Google the only major cloud that has low-latency single-zone object storage, standard regional object storage, and transparently-replicated dual-region object storage - all with the same API. For infra systems, this is great: code against the GCS API, and let the user choose the cost/latency/durability tradeoffs that make sense for their use case.
> This makes Google the only major cloud that has low-latency single-zone object storage, standard regional object storage, Absurd claim. S3 Express launched last year.
Sure, but AFAIK S3’s multi-region capabilities are quite far behind GCS’s.
S3 offers some multi-region replication facilities, but as far as I’ve seen they all come at the cost of inconsistent reads - which greatly complicates application code. GCS dual-region buckets offer strongly consistent metadata reads across multiple regions, transparently fetch data from the source region where necessary, and offer clear SLAs for replication. I don’t think the S3 offerings are comparable. But maybe I’m wrong - I’d love more competition here!
https://cloud.google.com/blog/products/storage-data-transfer...
Re: Colossus for Rapid Storage
#96This link makes so much more sense than the previous link did.
SSDs with high random I/o speeds are a significant contributor to the advantage. I think 20m writes per second are likely distributed over a network of drives to make that kind of speed possible.
Re: Colossus for Rapid Storage
#97Very cool! This makes Google the only major cloud that has low-latency single-zone object storage, standard regional object storage, and transparently-replicated dual-region object storage - all with the same API. For infra systems, this is great: code against the GCS API, and let the user choose the cost/latency/durability tradeoffs that make sense for their use case.
? s3: https://aws.amazon.com/pm/serv-s3 s3 express: https://aws.amazon.com/s3/storage-classes/express-one-zone/ cross-region replication: https://docs.aws.amazon.com/AmazonS3/latest/userguide/replic...
The cross-region replication I’ve seen for S3 (including the link you’ve provided) is fundamentally different from a dual-region GCS bucket. AWS is providing a way to automatically copy objects between distinct buckets, while GCS is providing a single bucket that spans multiple regions.
It’s much, much easier to code against a dual-region GCS bucket because the bucket namespace and object metadata are strongly consistent across regions.
Re: Colossus for Rapid Storage
#98Is this related at all the the private invite only anywhere caches? (or maybe they're GA now?) https://cloud.google.com/storage/docs/anywhere-cache
Anywhere Cache and Rapid Storage share some infrastructure inside of GCS and both are good solutions for improving GCS performance, but Anywhere Cache is an SSD cache in front of the normal buckets while Rapid Storage is a new type of bucket.
(I work on Google storage)
Re: Colossus for Rapid Storage
#99Earlier quoted context omitted.
I think the joke here is that somehow management refused to sell Colossus (which is such an obvious nice product just like BigQuery) before and it takes "AI" to convince them.
> which is such an obvious nice product just like BigQuery I always assumed (from outside Google) that the problem was that Colossus had to make a "no malicious actors" assumption in its design in order to make the performance/scaling guarantees it does; and that therefore just exposing it directly to the public would make it possible for someone to DoS-attack the Colossus cluster. My logic was that there's actually…
> a "no malicious actors" assumption in its design in order to make the performance/scaling guarantees it does
Didn't think deep into it, could this be solved with billing designs with more nuance?
Re: Colossus for Rapid Storage
#100Very cool! This makes Google the only major cloud that has low-latency single-zone object storage, standard regional object storage, and transparently-replicated dual-region object storage - all with the same API. For infra systems, this is great: code against the GCS API, and let the user choose the cost/latency/durability tradeoffs that make sense for their use case.
> This makes Google the only major cloud that has low-latency single-zone object storage, standard regional object storage, Absurd claim. S3 Express launched last year.
Isn't S3 Express not the same API? You have to use a "directory bucket" which isn't an object store anymore, as it has actual directories.
To be honest I'm not actually sure how different the API is. I've never used it. I just frequently trip over the existence of parallel APIs for directory buckets (when I'm doing something niche, mostly; I think GetObject/PutObject are the same.)