Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In S3 simplicity is table stakes

allthingsdistributed.com

Post reply on HN
Print view View original

91–100 of 142 posts

Re: In S3 simplicity is table stakes

#91

Post by rglover »

flessner wrote: post #6
neom wrote: post #3

little history: When we were getting ready to do an API at DigitalOcean I got asked "uhm... how should it feel?" I thought about that for about 11 seconds and said "if all our APIs feel as good as S3, it should be fine" - it's a good API.

The API, absolutely. It's only sad that the SDKs are often on the heavy side, I remember that the npm package used to be multiple megabytes as it bundled large parts of the core AWS SDK. Nowadays, I believe it's still around half a megabyte.

The footprint of the JS SDK is much better as they split all of it into service-specific packages, but the SDK APIs are a bit confusing (everything is a class that has to be instantiated—even config).

Re: In S3 simplicity is table stakes

#92

Post by icedchai »

alberth wrote: post #26

S3 is the simplest CRUD app you could create. It's essentially just the 4 functions of C.R.U.D done to a file. Most problems in tech are not that simple. Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).

Now add versioning, replication, logging, encryption, ACLs, CDN integration, event triggering (Lambda). I could go on. These are just some other features I can name off the top of my head. And it all has to basically run with zero downtime, 24x7...

Re: In S3 simplicity is table stakes

#94

Post by diroussel »

It's great that they added iceberg support I guess, but it's a shame that they also removed S3 Select. S3 Select wasn't perfect. For instance the performance was no where near as good as using DuckDB to scan a parquet file, since duck is smart, and S3 Select does a full table scan.

But S3 Select is nearly way cheaper that the new iceberg support. So if your needs are only for reading one parquet snapshot, we no need to do updates, then this change is not welcome.

Great article though, and I was pleased to see this at the end:

> We’ve invested in a collaboration with DuckDB to accelerate Iceberg support in Duck,

Re: In S3 simplicity is table stakes

#96

Post by CobrastanJorji »

> When we moved S3 to a strong consistency model, the customer reception was stronger than any of us expected.

This feels like one of those Apple-like stories about inventing and discovering an amazing, brand new feature that delighted customers but not mentioning the motivating factor of the competing products that already had it. A more honest sentence might have been "After years of customers complaining that the other major cloud storage providers had strong consistency models, customers were relieved when we finally joined the party."

Re: In S3 simplicity is table stakes

#97

Post by oblio »

achierius wrote: post #86
merb wrote: post #18

Earlier quoted context omitted.

The durability is not so good when you have a lot of objects

Why not? I don't work with web-apps or otherwise use object stores very often, but naively I would expect that "my objects not disappearing" would be a good thing.

I think their point is that you'd need even higher durability. With millions of objects, even 5+ nines means that you lose objects relatively constantly.

Re: In S3 simplicity is table stakes

#98

Post by gamegoblin »

huntaub wrote: post #41
graemep wrote: post #39

Earlier quoted context omitted.

How does that compare to competitors and things like distributed file systems?

I generally see object storage systems advertise 11 9s of availability. You would usually see a commercial distributed file system (obviously stuff like Ceph and Lustre will depend on your specific configuration) advertise less (to trade off performance for durability).

In general if you actually do the erasure coding math, almost all distributed storage systems that use erasure coding will have waaaaay more than 11 9s of theoretical durability

S3's original implementation might have only had 11 9s, and it just doesn't make sense to keep updating this number, beyond a certain point it's just meaningless

Like "we have 20 nines" "oh yeah, well we have 30 nines!"

To give an example of why this is the case, if you go from a 10:20 sharding scheme to a 20:40 sharding scheme, your storage overhead is roughly the same (2x), but you have doubled the number of nines

So it's quite easy to get a ton of theoretical 9s with erasure coding

Re: In S3 simplicity is table stakes

#99

Post by 1a527dd5 »

StratusBen wrote: post #95

For those interested in S3 Tables which is referenced in this blog post, we literally just published this overview on what they are and cost considerations of them that people might find interesting: https://www.vantage.sh/blog/amazon-s3-tables

https://www.vantage.sh/blog/amazon-s3-tables#s3-tables-cost

I can't make head or tails of the beginning of this sentence:-

> Pricing for S3 Tables is all and all not bad.

Otherwise lovely article!

Re: In S3 simplicity is table stakes

#100

Post by hobo_in_library »

angulardragon03 wrote: post #84
_fat_santa wrote: post #11

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 1…

I did a GCP training a while back, and the anecdote from one of the trainers was that the Cloud Storage team (GCP’s S3-compatible product) hadn’t lost a single byte of data since GCS had existed as a product. Crazy at that scale.

Eh, they have lost a bit
Post reply on HN