121–129 of 129 posts
Earlier quoted context omitted.
Of what you mentioned, only backblaze is similar (object storage with S3-like API), all others are apples to oranges.
You don't need very many terabytes to cover the labor cost of installing and maintaining an S3-compatible server program.
Earlier quoted context omitted.
There are some semantic differences compared to POSIX filesystems. A couple big ones: - You can only append to an object, and each object can only have one writer at the time. This is useful for distributed systems - you could have one process adding records to the end of a log, and readers pulling new records from the end. - It's also possible to "finalize" an object, meaning that it can't be appended to any more. (…
Why would you wish for a system with constraints like that, which other systems don't have?
Earlier quoted context omitted.
> Sure, but AFAIK S3’s multi-region capabilities are quite far behind GCS’s. Entirely different claim.
I claimed that Google is the only major cloud provider with all three of: - single-zone object storage buckets - regional object storage buckets - transparently replicated, dual region object storage buckets I agree that AWS has two of the three. AFAIK AWS does not have multi-region buckets - the closest they have is canned replication between single-region buckets.
Earlier quoted context omitted.
Clicking yourself a Bucket takes 5 Minutes. Building a Server and keeping it secure and up-to-date and fixing hardware issues, takes relevant time
Not to mention that I can: - Create a bucket and store 1MB in it without any overhead - Create 50 buckets with strong perimeters around them such that someone deleting the entire account doesn’t bring down the other 49 - Create a bucket and fill it with terabytes of data within seconds and don’t need to wait for hardware to be racked and stacked - Create a bucket, fill it with 2TB of data, and delete it tomorrow Clou…
For example, I agree that elasticity is great. But at the same time, to me, it sounds like bad engineering. Why do you need to store terabytes of data and then delete it - couldn't it be processed continuously, streamed, compressed, process changes only, and so on. A lot of engineering today is incredibly wasteful. Maybe your data source doesn't care, and just provides you with terabyte csv files, and you have no choice, but for engineers that care about efficiency, it reeks.
It might make a lot of sense in a highly corporate context where everything is hard, nobody cares, and the cost of inefficiency is just passed on to the customer (i.e. often government and tax payers). But the real problem here is that customers aren't demanding more efficiency.
Hats off to whoever convinced management that selling Colossus via cloud was Artificial Intelligence. Bravo.
Earlier quoted context omitted.
Why would you wish for a system with constraints like that, which other systems don't have?
Other systems don't offer the performance that Colossus offers, is why. POSIX has all kinds of silly features that aren't really necessary for every use case. Throwing away things like atomic writes by multiple writers allows the whole system to just go faster.
Earlier quoted context omitted.
Not to mention that I can: - Create a bucket and store 1MB in it without any overhead - Create 50 buckets with strong perimeters around them such that someone deleting the entire account doesn’t bring down the other 49 - Create a bucket and fill it with terabytes of data within seconds and don’t need to wait for hardware to be racked and stacked - Create a bucket, fill it with 2TB of data, and delete it tomorrow Clou…
I suspect the problem is that we're engineers in domains that have very different needs. For example, I agree that elasticity is great. But at the same time, to me, it sounds like bad engineering. Why do you need to store terabytes of data and then delete it - couldn't it be processed continuously, streamed, compressed, process changes only, and so on. A lot of engineering today is incredibly wasteful. Maybe your dat…
And plenty of use cases have natural growth. I do not throw away my pictures for example.
Data also grows dependent of users. More users, more 'live' data.
We have such a huge advantage with digital, we need to stop thinking its wasteful. Everything we do digital (pictures, finance data, etc.) is so much more energy and space efficient than what we had 20 years ago, we should just not delete data because we feel its wasteful.
Earlier quoted context omitted.
I don't fault them for this at all. AI isn't possible without the full infra stack, which clearly includes storage (and compute, and networking, and data pipelining, and and and...). There's an entire ecosystem of ISVs that only do one of these things, very well (Pure Storage, for example, or Lamba or Coreweave, or Confluent (Kafka + Flink with LLM integration). While it might be more precisely accurate to state "AI…
I think the joke here is that somehow management refused to sell Colossus (which is such an obvious nice product just like BigQuery) before and it takes "AI" to convince them.
I don't know that it took "AI branding" to convince anybody. I think these workloads potentially enabled additional demand/market for such a product that may not have been there before.
One of the challenges with exposing native Colossus was always that it's just different enough from how people elsewhere are used to use Storage that there was a lot of uncertainty about the addressable market of a "native" Colossus offering. It's not a POSIX file system. Some of the specific differences (eg. no random writes) are part of what makes Colossus powerful and performant on HDDs, but it means you have to write your application to work well within its constraints. Google has been doing that for a long time. If you haven't, even if it's an amazing product, is it worth rewriting your applications or middleware?
Rapid Storage basically addresses this by adding the object store API on top if it (TIL from this thread that there's a lower abstraction client in the works as well).
Anyway, the team behind this is awesome. Awesome tech, awesome people. Seeing this launched at Next and seeing some appreciation on HN makes me very grateful.
Earlier quoted context omitted.
Other systems don't offer the performance that Colossus offers, is why. POSIX has all kinds of silly features that aren't really necessary for every use case. Throwing away things like atomic writes by multiple writers allows the whole system to just go faster.
It sounds like you have to find a design that meets your performance target and usage patterns - just like anything else. It also sounds like Google's CFS is a grass is greener situation - you heard Google had something that solved the problem you have, so you want it. But the reason it sounds good, compared to the other designs, is that you haven't had to actually use it and run into its quirks yet.
On the other hand, if you've been building your applications against expectations of different semantics (like POSIX), retrofitting this into your existing application is really hard, and potentially awkward. This is (IMO) why there hasn't been an overtly Colossus based Google Cloud offering previously. (Though it's well publicized that both Persistent Disk and GCS use Colossus in their implementation.)
One of the reasons why it would be extremely hard to just set up or build CFS elsewhere or on a different abstraction level is that while it may look quite achievable to implement the high level architecture, there is vast complexity in the practical implementation side. The tremendous user isolation it affords for an MT system, the resilience it has against various types of failures and high throughput planned maintenance, the specialization it and its dependencies have to use specific hardware optimally.
(I work on Google storage part time, I am not a Colossus developer.)