Minimizing S3 API costs with distributed MMAP

#1

Post by richieartoul » Wed, Oct 11, 2023, 4:08 PM UTC

Minimizing S3 API costs with distributed MMAP
warpstream.com

Re: Minimizing S3 API costs with distributed MMAP

#2

Post by refset » Wed, Oct 11, 2023, 4:55 PM UTC

This is a really interesting design - kudos.

> Compacting the data may sound expensive, but in practice it's highly efficient since it only needs to happen once

Is there any handling for tombstone records?

Re: Minimizing S3 API costs with distributed MMAP

#3

Post by ryanworl » Wed, Oct 11, 2023, 5:01 PM UTC

refset wrote: post #2

This is a really interesting design - kudos. > Compacting the data may sound expensive, but in practice it's highly efficient since it only needs to happen once Is there any handling for tombstone records?

[WarpStream co-founder]

WarpStream doesn't implement compacted topics today. It is on our roadmap, though. Compacted topics are typically not used in high-throughput workloads, so our plan is to delay compactions for longer than a disk-based system would to trade space amplification for write amplification.

Re: Minimizing S3 API costs with distributed MMAP

#4

Post by refset » Wed, Oct 11, 2023, 8:09 PM UTC

ryanworl wrote: post #3

refset wrote: post #2

This is a really interesting design - kudos. > Compacting the data may sound expensive, but in practice it's highly efficient since it only needs to happen once Is there any handling for tombstone records?

[WarpStream co-founder] WarpStream doesn't implement compacted topics today. It is on our roadmap, though. Compacted topics are typically not used in high-throughput workloads, so our plan is to delay compactions for longer than a disk-based system would to trade space amplification for write amplification.

Thanks for clarifying.

> Compacted topics are typically not used in high-throughput workloads

TIL, but it makes sense. Compaction/retention policies certainly introduce a lot of extra tradeoffs dimensions.