Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In S3 simplicity is table stakes

allthingsdistributed.com

Post reply on HN
Print view View original

61–70 of 142 posts

Re: In S3 simplicity is table stakes

#61

Post by bob1029 »

I really enjoy using S3 to serve arbitrary blobs. It perfectly solves the problem space for my use cases.

I avoid getting tangled in authentication mess by simply naming my files using type4 GUIDs and dumping them in public buckets. The file name is effectively the authentication token and expiration policies are used to deal with the edges.

This has been useful for problems like emailing customers gigantic reports and transferring build artifacts between systems. Having a stable URL that "just works" everywhere easily pays for the S3 bill in terms of time & frustration saved.

Re: In S3 simplicity is table stakes

#62

Post by jjice »

ipsento606 wrote: post #54

Lots of comments here talking about how great S3 is. Anyone willing to give a cliff notes about what's good about it? I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.

S3 has often fallen into a "catch all" solution for me whenever I need to store data large enough that I don't want to keep it in a database (RDBMS or Redis).

Need to save a file somewhere? Dump it in S3. It's generally affordable (obviously dependent on scale and use), fast, easy, and super configurable.

Being able to expose something to the outside, or with a presigned URL is a huge advantage as well.

Off the top of my head, I think of application storage generally in this tier ordering (just off the top of my head based on the past few years of software development, no real deep thought here):

1. General application data that needs to be read, written, and related - RDBMS

2. Application data that needs to be read and written fast, no relations - Redis

3. Application data that is mostly stored and read - S3

Replace any of those with an equivalent storage layer.

Re: In S3 simplicity is table stakes

#63

Post by ak217 »

ipsento606 wrote: post #54

Lots of comments here talking about how great S3 is. Anyone willing to give a cliff notes about what's good about it? I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.

One underappreciated feature of S3 - that allowed it to excel in workloads like the Tables feature described in the article - is that it's able to function as the world's highest throughout network filesystem. And you don't have to do anything to configure it (as the article points out). By storing data on S3, you get to access the full cross-sectional bandwidth of EC2, which is colossal. For effectively all workloads, you will max out your network connection before S3's. This enables workloads that can't scale anywhere else. Things like data pipelines generating unplanned hundred-terabit-per-second traffic spikes with hotspots that would crash any filesystem cluster I've ever seen. And you don't have to pay a lot for it- once you're done using the bandwidth, you can archive the data elsewhere or delete it.

Re: In S3 simplicity is table stakes

#64

Post by aeyes »

user9999999999 wrote: post #43

if only metadata could be queried without processing a csv output file first, imagine storing thumbnails in there even! copied objects had actual events, not something you have to dig cloudtrail for, you could get last update time from a bucket to make caching easier

When S3 Tables launched they made the Metadata available using this technology. So you can query it like an Apache Iceberg table.

https://aws.amazon.com/blogs/aws/introducing-queryable-objec...

Re: In S3 simplicity is table stakes

#65

Post by huntaub »

ak217 wrote: post #63
ipsento606 wrote: post #54

Lots of comments here talking about how great S3 is. Anyone willing to give a cliff notes about what's good about it? I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.

One underappreciated feature of S3 - that allowed it to excel in workloads like the Tables feature described in the article - is that it's able to function as the world's highest throughout network filesystem. And you don't have to do anything to configure it (as the article points out). By storing data on S3, you get to access the full cross-sectional bandwidth of EC2, which is colossal. For effectively all workload…

You've totally hit the nail on the head. This is the real moat of S3, the fact that they have so much front-end throughput available from the gigantic buildout that folks can take advantage of without any capacity pre-planning.

Re: In S3 simplicity is table stakes

#66

Post by golly_ned »

alberth wrote: post #26

S3 is the simplest CRUD app you could create. It's essentially just the 4 functions of C.R.U.D done to a file. Most problems in tech are not that simple. Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).

A file system is simple. Open, read, close. Most tech problems are not that simple. How hard could a filesystem be?

Re: In S3 simplicity is table stakes

#67

Post by jorams »

arnath wrote: post #34

I found out last year that you can actually run a full SPA using S3 and a CDN. It’s kind of a nuts platform

Since everything you need to run "a full SPA" is to serve some static files over an internet connection I'm not sure how that tells you anything interesting about the platform. It's basically the simplest thing a web server can do.

Re: In S3 simplicity is table stakes

#68

Post by nerdjon »

ipsento606 wrote: post #54

Lots of comments here talking about how great S3 is. Anyone willing to give a cliff notes about what's good about it? I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.

There are a few things about S3 that I find extremely powerful.

The biggest is, if I need to store some data. I know what the data is (so I don't need to worry about needing to in a moment notice traverse a file structure for example, I know my filenames), I can store that data, I don't need to figure out how much space I need ahead of time and it is there when I need it. Maybe it automatically moves to another storage tier to save me some money but I can reliably assume it will be there when I need it. Just that simplicity alone is worth a lot, I never need to think later that I need to expand some space, possibly introducing downtime depending on the setup, maybe dealing with partitions, etc.

Related to that is static hosting. I have ran a CDN and other static content out of S3 with cloud front in front of it. The storage cost was almost non existent due to how little actual data we were talking about and only paid for cloudfront costs when there were requests. If nothing was being used it was almost "free". Even when being used it was very cheap for my use cases.

Creating daily inventory reports in S3 is awesome.

But the thing that really is almost "magic" once you understand its quirks. Athena (and quick sight built on top of that and similar tools). The ability to store data in S3 like inventory reports that I already mentioned, access logs, cloud watch logs, or any structured data that you may not need to query often enough to warrant a full long running database. It may cost you a few dollars to run your Athena query and it is not going to be super quick, but if you know what you're looking for it is amazing.

Re: In S3 simplicity is table stakes

#69

Post by bityard »

bob1029 wrote: post #61

I really enjoy using S3 to serve arbitrary blobs. It perfectly solves the problem space for my use cases. I avoid getting tangled in authentication mess by simply naming my files using type4 GUIDs and dumping them in public buckets. The file name is effectively the authentication token and expiration policies are used to deal with the edges. This has been useful for problems like emailing customers gigantic reports a…

My favorite use case for S3 API-compatible solutions: I often run into systems that generate lots of arbitrary data that only have temporary importance. A common example might be intermediate build artifacts, or testing ephemera (browser screenshots, etc). Things that are needed for X number of months and then just need to disappear.

Yeah, we can dump those to a filesystem. But then we have to ask which filesystem? What should the directory layout be? If there are millions or billions of objects, walking the whole tree gets expensive. Do we write a script to clean everything up? Run it via cron or some other job runner?

With S3, you just write your artifact to S3 with a TTL and it gets deleted automagically when it should. No cron jobs, no walking the whole tree. And you can set up other lifecycle options if you need it moved to other (cheaper) storage later on, backups, versioning, and whatever else.

For on-prem, you have Minio, Garage, or SeaweedFS. These are pretty nice to deploy the servers however you need for the level of reliability/durability you require.

Re: In S3 simplicity is table stakes

#70

Post by bentobean »

It's funny—S3 started as a "simple" storage service, and now it's handling entire table abstractions. Reminds me how SQL was declared dead every few years, yet here we are, building ever more complex data solutions on top of supposedly simple foundations.
Post reply on HN