Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Express Is All You Need

warpstream.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: S3 Express Is All You Need

#71

Post by haimez »

parhamn wrote: post #3
BonoboIO wrote: post #2

Has anyone here a usecase which would perform better with this new S3 Express Tier? And a second question, would it be worth the 8x times surcharge?

I think the key benefit brushed on by this article is the potential 10x improvement in access speeds (which has many applications, beyond reducing your s3 op charges). > S3 Express One Zone can improve data access speeds by 10x and reduce request costs by 50% compared to S3 Standard and scales to process millions of requests per minute.

10x reduction in latency, higher storage costs with lower access costs (SSD instead of spinning disks). So high I/O, small files situations (with no need for cross AZ access) are where the benefits can be found.

Re: S3 Express Is All You Need

#72

Post by fulmicoton »

kernelsanderz wrote: post #69
francoismassot wrote: post #42

We tested S3 Express for our search engine quickwit [0] a couple of weeks ago. While this was really satisfying on the performance side, we were a bit disappointed by the price, and I mostly agree with the article on this matter. I can see some very specific use cases where the pricing should be OK but currently, I would say most of our users will just stay on the classic S3 and add some local SSD caching if they hav…

I'd be fascinated if you could share your insights from using this. Where does the pricing fall down? And is the latency/throughput a big improvement for this use case? (ie. externalizing a search index).

I ran the benchmark at Quickwit. I confirm it works as intended. I was extremely excited about this feature, primarily interested in the decreased GET request cost, and secondly the lower latency.

Unfortunately the price model puts it in a place where it is the right technology only for some very rare places.

In a nutshell the key thing you need to know is: - The storage is 6.4x expensive than classic S3. - The GET requests are 2x cheaper (with additional cost for large requests). - Your data is replicated within a single region. - latency is single digit ms.

From a pure cost wise point of view, the realm where it makes sense to use it is there, but small, and often competes more with EBS than it competes with S3.

Re: S3 Express Is All You Need

#73

Post by nojvek »

edmundsauto wrote: post #68
fswd wrote: post #33

I solved this problem locally. When uploading a file to the server before going to S3 it is cached in redis. Whenever the codebase needs to use the file, it checks redis, and if it is not there it fetches it and caches it again.

So you store the actual image data in redis? That's interesting, no issues with storing binary data? I ask because I've always been taught to not store files in a database. This use case sounds like an interesting exception.

Files are just a bunch of bytes. No harm in putting them in a database.

There were some benchmarks, I couldn’t fine where SQLite was faster than native file system at retrieving, searching and adding files to a large directory.

Re: S3 Express Is All You Need

#74

Post by kernelsanderz »

fulmicoton wrote: post #72
kernelsanderz wrote: post #69

Earlier quoted context omitted.

I'd be fascinated if you could share your insights from using this. Where does the pricing fall down? And is the latency/throughput a big improvement for this use case? (ie. externalizing a search index).

I ran the benchmark at Quickwit. I confirm it works as intended. I was extremely excited about this feature, primarily interested in the decreased GET request cost, and secondly the lower latency. Unfortunately the price model puts it in a place where it is the right technology only for some very rare places. In a nutshell the key thing you need to know is: - The storage is 6.4x expensive than classic S3. - The GET r…

thank so much for sharing. Amazing product BTW!

Re: S3 Express Is All You Need

#75

Post by seasily »

kernelsanderz wrote: post #65
Sirupsen wrote: post #12

Most production storage systems/databases built on top of S3 spend a significant amount of effort building an SSD/memory caching tier to make them performant enough for production (e.g. on top of RocksDB). But it's not easy to keep it in sync with blob... Even with the cache, the cold query latency lower-bound to S3 is subject to ~50ms roundtrips [0]. To build a performant system, you have to tightly control roundtri…

Very excited about being able to build scalable vector databases on DiskANN like turbopuffer or lancedb. These changes in latency are game changing. The best server is no server. The capability a low latency vector database application that runs in lambda and S3 and is dirt cheap is pretty amazing.

The clear use case is serverless—without the complications of DynamoDB (expensive, 0.03/GB read), DynamoDB+DAX (VPC complications), or Redis (again, VPC requirements).

This instantly makes a number of applications able to run directly on S3, sans any caching system.

Re: S3 Express Is All You Need

#76

Post by __turbobrew__ »

Sirupsen wrote: post #12

Most production storage systems/databases built on top of S3 spend a significant amount of effort building an SSD/memory caching tier to make them performant enough for production (e.g. on top of RocksDB). But it's not easy to keep it in sync with blob... Even with the cache, the cold query latency lower-bound to S3 is subject to ~50ms roundtrips [0]. To build a performant system, you have to tightly control roundtri…

Similar to vector databases this could be really useful for hosting cloud optimized geotiffs for mapping purposes. At a previous job we were able to do on the fly tiling in about 100ms but with this new storage class you could probably make something that could tile just as fast or even faster than arcgis with all of its proprietary optimizations and goop.

Take it a step further, gdal supports s3 raster data sources out of the box for a while now. Any gdal powered system may be able to operate on s3 files as if they are local.

Re: S3 Express Is All You Need

#77

Post by avinassh »

jamiesonbecker wrote: post #35
fswd wrote: post #33

I solved this problem locally. When uploading a file to the server before going to S3 it is cached in redis. Whenever the codebase needs to use the file, it checks redis, and if it is not there it fetches it and caches it again.

Exactly. Write-through cache is exactly how Userify[0] used to work for self-hosted versions. (when it was Python, we used Redis to keep state synced across multiple processes, but now that it's a Go app, we do all the caching and state management in memory using Ristretto[1]) However, we now install by default to local disk filesystem, since it's much faster to just do a periodic S3 hot sync, like with restic or aws…

What were the reasons to move from Redis to Ristretto? Both seem to be very different, since Redis is distributed where as Ristretto is local to the process.

Re: S3 Express Is All You Need

#78

Post by avinassh »

nojvek wrote: post #73
edmundsauto wrote: post #68

Earlier quoted context omitted.

So you store the actual image data in redis? That's interesting, no issues with storing binary data? I ask because I've always been taught to not store files in a database. This use case sounds like an interesting exception.

Files are just a bunch of bytes. No harm in putting them in a database. There were some benchmarks, I couldn’t fine where SQLite was faster than native file system at retrieving, searching and adding files to a large directory.

SQLite reads and writes small blobs (for example, thumbnail images) 35% faster¹ than the same blobs can be read from or written to individual files on disk using fread() or fwrite().

Furthermore, a single SQLite database holding 10-kilobyte blobs uses about 20% less disk space than storing the blobs in individual files.

The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once, whereas open() and close() are invoked once for each blob when using blobs stored in individual files. It appears that the overhead of calling open() and close() is greater than the overhead of using the database. The size reduction arises from the fact that individual files are padded out to the next multiple of the filesystem block size, whereas the blobs are packed more tightly into an SQLite database.

https://www.sqlite.org/fasterthanfs.html

Re: S3 Express Is All You Need

#79

Post by jamiesonbecker »

avinassh wrote: post #77
jamiesonbecker wrote: post #35

Earlier quoted context omitted.

Exactly. Write-through cache is exactly how Userify[0] used to work for self-hosted versions. (when it was Python, we used Redis to keep state synced across multiple processes, but now that it's a Go app, we do all the caching and state management in memory using Ristretto[1]) However, we now install by default to local disk filesystem, since it's much faster to just do a periodic S3 hot sync, like with restic or aws…

What were the reasons to move from Redis to Ristretto? Both seem to be very different, since Redis is distributed where as Ristretto is local to the process.

In our case, Python (because of the GIL) required us to have a single python process per core in order to take advantage of multiple cores, and so we needed Redis to maintain a unified memory state across all the cores, but Go can automatically span across multiple cores.

We also saw about a 10x speedup by moving all caching into the server process, and since it was all in the same process, we no longer had to compress and encrypt data before sending to Redis. We still checkpoint the moving server state, encrypted and compressed, to disk every sixty seconds, just like Redis would do with BGSAVE, so we can start back up within a few seconds (actually faster than the old Redis after a restart.)

Re: S3 Express Is All You Need

#80

Post by KTibow »

kristianp wrote: post #59

I saw "X is all you Need" with the "Attention is all you need" paper [1], which launched the Transformer upon the world. Is it the first instance of that phrase? [1] https://arxiv.org/abs/1706.03762

There was a SIGBOVIK paper on this
Post reply on HN