Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Express Is All You Need

warpstream.com

Post reply on HN
Print view View original

31–40 of 87 posts

Re: S3 Express Is All You Need

#31

Post by dpedu »

sparrc wrote: post #30
throwitaway222 wrote: post #13

I don't understand why EFS never gets major shout outs - it's way better than S3: systems can mount it as a drive, shared across systems, already has had super low latency... Not sure what s3 express is really useful for if EFS already exists.

In my experience the biggest drawback with EFS is startup time for systems that mount it in. For example a container or EC2 instance might only need a tiny bit of your storage and with s3 can just download what it needs when it needs it. As opposed to EFS where the container or instance needs to load in the entire datastore on startup which can add minutes to startup time if the EFS drive is large.

My understanding is that EFS is exposed as an NFS share. I haven't used it personally, but NFS mounting is generally fast, nearly instant. What does "load in the entire datastore" mean?

Re: S3 Express Is All You Need

#32

Post by tneely »

yeeeloit wrote: post #26
throwitaway222 wrote: post #13

I don't understand why EFS never gets major shout outs - it's way better than S3: systems can mount it as a drive, shared across systems, already has had super low latency... Not sure what s3 express is really useful for if EFS already exists.

I wonder if Mountpoint for S3 along with this new Express option makes it a direct competitor to EFS for some use cases. https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountp...

I'm quite curious about this too - both from a cost and performance perspective. If S3 Express is close enough to EFS on these metrics, then I'd say it wins out due to the sheer ubiquity and portability of S3 these days.

Re: S3 Express Is All You Need

#33

Post by fswd »

I solved this problem locally. When uploading a file to the server before going to S3 it is cached in redis. Whenever the codebase needs to use the file, it checks redis, and if it is not there it fetches it and caches it again.

Re: S3 Express Is All You Need

#34

Post by ericpauley »

dpedu wrote: post #31
sparrc wrote: post #30

Earlier quoted context omitted.

In my experience the biggest drawback with EFS is startup time for systems that mount it in. For example a container or EC2 instance might only need a tiny bit of your storage and with s3 can just download what it needs when it needs it. As opposed to EFS where the container or instance needs to load in the entire datastore on startup which can add minutes to startup time if the EFS drive is large.

My understanding is that EFS is exposed as an NFS share. I haven't used it personally, but NFS mounting is generally fast, nearly instant. What does "load in the entire datastore" mean?

EFS mounting is definitely nearly instant. I use it constantly.

Re: S3 Express Is All You Need

#35

Post by jamiesonbecker »

fswd wrote: post #33

I solved this problem locally. When uploading a file to the server before going to S3 it is cached in redis. Whenever the codebase needs to use the file, it checks redis, and if it is not there it fetches it and caches it again.

Exactly. Write-through cache is exactly how Userify[0] used to work for self-hosted versions. (when it was Python, we used Redis to keep state synced across multiple processes, but now that it's a Go app, we do all the caching and state management in memory using Ristretto[1])

However, we now install by default to local disk filesystem, since it's much faster to just do a periodic S3 hot sync, like with restic or aws-cli, than to treat S3 as the primary backing store, or just version the EBS or instance volume. The other reason you might want to use S3 as a primary is if you use a lot of disk, but our files are compressed and extremely small, even for a large installation with tens of thousands of users and instances.

0. https://userify.com (ssh key management + sudo for teams)

1. https://github.com/dgraph-io/ristretto

Re: S3 Express Is All You Need

#36

Post by YetAnotherNick »

awoimbee wrote: post #21
paulddraper wrote: post #11

Earlier quoted context omitted.

A cache with large blobs (images, etc)

If it's only a cache it should be on EBS, which is still way faster and 2x less expensive. I started a migration to s3 for such a project (container image caching) but then stopped when I realized what I was doing.

Yes, EBS is the gold standard but managing a EBS to scale up and down instantly, be available to multiple instances, lifecycle management, managing replica, switchover etc. are definitely not easy. And EBS are bad choice when throughput needed is very spiky.

Re: S3 Express Is All You Need

#37

Post by saddlerustle »

gchamonlive wrote: post #18
richieartoul wrote: post #16

Earlier quoted context omitted.

Do you have any more details you can share about the performance of EFS? I've never met anyone who has actually used it in anger.

Throughput scales with the amount of data in it, it is in the docs. So depending on the application, even if latency is better, the speeds are atrocious at lower volumes of persisted data.

That’s not true anymore with EFS Elastic Throughput

Re: S3 Express Is All You Need

#38

Post by amarshall »

influx wrote: post #8
tjoff wrote: post #4

> However, the new storage class does open up an exciting new opportunity for all modern data infrastructure: the ability to tune an individual workload for low latency and higher cost or higher latency and lower cost with the exact same architecture and code. I get it, but at the same time that is also what you lost when you locked yourself in with a particular vendor.

There's not much to the S3 API, and data import/export even at massive scale is available with Snowball. Sure, there's many other AWS services that aren't available at other vendors, but blob storage is commodified at this point.

Exporting data from S3 is ludicrously expensive, even with Snowball it’s $30/TB just for network egress.

Re: S3 Express Is All You Need

#39

Post by dekhn »

jamesblonde wrote: post #24
Sirupsen wrote: post #12

Most production storage systems/databases built on top of S3 spend a significant amount of effort building an SSD/memory caching tier to make them performant enough for production (e.g. on top of RocksDB). But it's not easy to keep it in sync with blob... Even with the cache, the cold query latency lower-bound to S3 is subject to ~50ms roundtrips [0]. To build a performant system, you have to tightly control roundtri…

We built HopsFS-S3 [0] for exactly this problem, and have running it as part of Hopsworks now for a number of years. It's a network-aware, write-through cache for S3 with a HDFS API. Metadata operations are performed on HopsFS, so you don't have the other problems list max listing operations return 1000 files/dirs. NVMe is what is changing the equation, not SSD. NVMe disks now have up to 8 GB/s, although the crap in…

the numbers you're giving are throughput (byte/sec) not latency.

The comment you reply to is talking mostly about latency - reporting that S3 object get latencies (time to open the object and return its head) in the single-digits ms, where S3 was 50ms before.

BTW EBS can do 4GB/sec per volume. But you will pay for it.

Re: S3 Express Is All You Need

#40

Post by dekhn »

candiddevmike wrote: post #14
throwitaway222 wrote: post #13

I don't understand why EFS never gets major shout outs - it's way better than S3: systems can mount it as a drive, shared across systems, already has had super low latency... Not sure what s3 express is really useful for if EFS already exists.

EFS is really expensive and has terrible latency with small files in my experience

When you set up EFS did you maximize the IO settings?

Before doing that it was unacceptably slow. After doing that it was unacceptably expensive.

Post reply on HN