Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In S3 simplicity is table stakes

allthingsdistributed.com

Post reply on HN
Print view View original

81–90 of 142 posts

Re: In S3 simplicity is table stakes

#81

Post by hadlock »

vb-8448 wrote: post #30
sjsdaiuasgdia wrote: post #22

Earlier quoted context omitted.

That's going to depend a lot on what your needs are, particularly in terms of redundancy and durability. S3 takes care of a lot of that for you. One server with a RAID array can survive, usually, 1 or maybe 2 drive failures. The remaining drives in the array will have to do more work when a failed drive is replaced and data is copied to the new array member. This sometimes leads to additional failures before replacem…

How many businesses or applications really need 99.999999999% durability and 99.99% availability? Is your whole stack organized to deliver the forementioned durability and availability?

There are a lot of companies who their livelihood depends on their proprietary data, and loss of that data would be a company-ending-event. I'm not sure how the calculus works out exactly, but having additional backups and types of backups to reduce risk is probably one of the smaller business expenses one can pick up. Sending a couple TB of data to three+ cloud providers on top of your physical backups is in the tens of dollars per month.

Re: In S3 simplicity is table stakes

#82

Post by timewizard »

_1tem wrote: post #10

I have a feeling that economies of scale have a point of diminishing returns. At what point does it become more costly and complicated to store your data on S3 versus just maintaining a server with RAID disks somewhere? S3 is an engineering marvel, but it's an insanely complicated backend architecture just to store some files.

> At what point does it become more costly and complicated to store your data on S3 versus just maintaining a server with RAID disks somewhere?

It's more costly immediately. S3 storage prices are above what you would pay even for triply redundant media and you have to pay for data transfer at a very high rate to both send and receive data to the public internet.

It's far less complicated though. You just create a bucket and you're off to the races. Since the S3 API endpoints are all public there's not even a delay for spinning up the infrastructure.

Where S3 shines for me is two things. Automatic lifecycle management. Objects can be moved in between storage classes based on the age of the object and even automatically deleted after expiration. The second is S3 events which are also _durable_ and make S3 into an actual appliance instead of just a convenient key/value store.

Re: In S3 simplicity is table stakes

#83

Post by hadlock »

ipsento606 wrote: post #54

Lots of comments here talking about how great S3 is. Anyone willing to give a cliff notes about what's good about it? I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.

Do you need to replace your SFTP server? S3. Do you need to backup TB of db files? S3. Do you need a high performance web cache? S3. Host your SPA? S3 backs cloudfront. Shared filesystem between desktop computers? Probably a bad idea but you can do it with S3. Need a way for customers to securely drop files somewhere? Signed S3 URI. Need to store metrics? Logs? S3. Load balancer logs? S3. And it's cheaper than an EBS volume, and doesn't need resizing every couple of quarters. And there are various SLAs which make it cheaper (Glacier) or more expensive (High Performance). S3 makes a great storage backend for a lot of use cases especially when your data is coming in from multiple regions across the globe. There are some quibbles about eventual consistency but in general it is an easy backend to build for.

Re: In S3 simplicity is table stakes

#84

Post by angulardragon03 »

_fat_santa wrote: post #11

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 1…

I did a GCP training a while back, and the anecdote from one of the trainers was that the Cloud Storage team (GCP’s S3-compatible product) hadn’t lost a single byte of data since GCS had existed as a product. Crazy at that scale.

Re: In S3 simplicity is table stakes

#85

Post by sebastiansm »

cruffle_duffle wrote: post #37
alberth wrote: post #26

S3 is the simplest CRUD app you could create. It's essentially just the 4 functions of C.R.U.D done to a file. Most problems in tech are not that simple. Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).

That’s when you really know you hid all the complexity well. When people call your globally replicated data store with granular permissions, sophisticated data retention policies, versioning, and manage to have, what, seven (ten?) nines or something, “simple”. No problem. I’m sure ChatGPT could cook up a replacement in a weekend. Like Dropbox it’s just rsync with some scripts that glue it together. How hard could it…

I could build a netflix in a weekend.

Re: In S3 simplicity is table stakes

#86

Post by achierius »

merb wrote: post #18
_fat_santa wrote: post #11

S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system. One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 1…

The durability is not so good when you have a lot of objects

Why not? I don't work with web-apps or otherwise use object stores very often, but naively I would expect that "my objects not disappearing" would be a good thing.

Re: In S3 simplicity is table stakes

#87

Post by szundi »

dekhn wrote: post #78
golly_ned wrote: post #66

Earlier quoted context omitted.

A file system is simple. Open, read, close. Most tech problems are not that simple. How hard could a filesystem be?

Locking, checks after unclean shutdown, sparse files, high performance, reliabilty.... are all things that make filesystems harder.

[dead]

Re: In S3 simplicity is table stakes

#88

Post by o10449366 »

alberth wrote: post #26

S3 is the simplest CRUD app you could create. It's essentially just the 4 functions of C.R.U.D done to a file. Most problems in tech are not that simple. Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).

[flagged]

Re: In S3 simplicity is table stakes

#89

Post by zerd »

_1tem wrote: post #10

I have a feeling that economies of scale have a point of diminishing returns. At what point does it become more costly and complicated to store your data on S3 versus just maintaining a server with RAID disks somewhere? S3 is an engineering marvel, but it's an insanely complicated backend architecture just to store some files.

One interesting thing about S3 is the vast scale of it. E.g. if you need to store 3 PB of data you might need 150 HDDs + redundancy, but if you store it on S3 it's chopped up and put on tens of thousands of HDDs, which helps with IOPS and throughput. Of course that's shared with others, which is why smart placement is key, so that hot objects are spread out.

Some details in https://www.allthingsdistributed.com/2023/07/building-and-op... / https://www.youtube.com/watch?v=sc3J4McebHE

Re: In S3 simplicity is table stakes

#90

Post by rglover »

alberth wrote: post #26

S3 is the simplest CRUD app you could create. It's essentially just the 4 functions of C.R.U.D done to a file. Most problems in tech are not that simple. Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).

I used to have the same opinion until I built my own CDN. Scaling something like that is no joke, let alone ensuring you handle consistency and caching properly.

A basic implementation is simple, but at S3 scale, that's a whole different ball game.

Post reply on HN