Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In S3 simplicity is table stakes

allthingsdistributed.com

Post reply on HN
Print view View original

11–20 of 142 posts

Re: In S3 simplicity is table stakes

#11

Post by _fat_santa »

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system.

One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 10,000 years. Since then anytime I talk about S3's durability I bring up that example and it always seems to convey the point for a layperson.

And it's "simplicity" is truly elegant. When I first started using S3 I thought of it as a dumb storage location but as I learned more I realized that it had some wild features that they all managed to hide properly so when you start it looks like a simple system but you can gradually get deeper and deeper until your S3 bucket is doing some pretty sophisticated stuff.

Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors.

Re: In S3 simplicity is table stakes

#12

Post by easton »

waiwai933 wrote: post #8

> I’ve seen other examples where customers guess at new APIs they hope that S3 will launch, and have scripts that run in the background probing them for years! When we launch new features that introduce new REST verbs, we typically have a dashboard to report the call frequency of requests to it, and it’s often the case that the team is surprised that the dashboard starts posting traffic as soon as it’s up, even befor…

Maybe this is a faster way of getting AWS feature requests heard.

I'm going to write a script that keeps trying to call ecs:MakeFargateCacheImages.

Re: In S3 simplicity is table stakes

#13

Post by apetresc »

ecshafer wrote: post #9
greatpostman wrote: post #4

I can guarantee you, nothing is simple about S3. I bet engineers spend months on a single configuration change to the underlying subsystems.

If you read the article, they say that exactly. Its by Dr Werner Vogels, they know exactly what goes into S3, since they are a principal engineer on the project.

It’s Vogels’ blog but this is a guest post by Andy Warfield.

Re: In S3 simplicity is table stakes

#14

Post by malfist »

flessner wrote: post #6
neom wrote: post #3

little history: When we were getting ready to do an API at DigitalOcean I got asked "uhm... how should it feel?" I thought about that for about 11 seconds and said "if all our APIs feel as good as S3, it should be fine" - it's a good API.

The API, absolutely. It's only sad that the SDKs are often on the heavy side, I remember that the npm package used to be multiple megabytes as it bundled large parts of the core AWS SDK. Nowadays, I believe it's still around half a megabyte.

I don't know if the npm is the same way, but the java sdk now has options by service. So you can include just s3 instead of all of aws

Re: In S3 simplicity is table stakes

#15

Post by Tanjreeve »

_1tem wrote: post #10

I have a feeling that economies of scale have a point of diminishing returns. At what point does it become more costly and complicated to store your data on S3 versus just maintaining a server with RAID disks somewhere? S3 is an engineering marvel, but it's an insanely complicated backend architecture just to store some files.

Probably never. The complexity is borne by Amazon. Even before any of the development begins if you want a RAID setup with some sort of decent availability you've already multiplied your server costs by the number of replicas you'd need. It's a Sisyphean task that also has little value for most people.

Much like twitter it's conceptually simple but it's a hard problem to solve at any scale beyond a toy.

Re: In S3 simplicity is table stakes

#16

Post by victorp13 »

_fat_santa wrote: post #11

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 1…

> Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors.

Counter-point: You know that you're the dominant player. See: .psd, .pdf, .xslx. Not particularly good file types, yet widely supported by competitor products.

Re: In S3 simplicity is table stakes

#17

Post by ajb »

easton wrote: post #12
waiwai933 wrote: post #8

> I’ve seen other examples where customers guess at new APIs they hope that S3 will launch, and have scripts that run in the background probing them for years! When we launch new features that introduce new REST verbs, we typically have a dashboard to report the call frequency of requests to it, and it’s often the case that the team is surprised that the dashboard starts posting traffic as soon as it’s up, even befor…

Maybe this is a faster way of getting AWS feature requests heard. I'm going to write a script that keeps trying to call ecs:MakeFargateCacheImages.

It could also be hackers, as when a new service launches is exactly when it will be most buggy. And the contents of S3 are a big payoff.

Re: In S3 simplicity is table stakes

#18

Post by merb »

_fat_santa wrote: post #11

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 1…

The durability is not so good when you have a lot of objects

Re: In S3 simplicity is table stakes

#19

Post by laluser »

_fat_santa wrote: post #11

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 1…

It’s designed for that level of durability, but it’s only as good as a single change or correlated set of hardware failures that can quickly change the theoretical durability model. Or even corrupting data is possible too.
Post reply on HN