Viewing profile — ryanworl

Does this trick preclude the ability to sort your data within a partition? You wouldn’t be able to rely on the row IDs being sequential anymore to be able to just refer to a prefix…

This is a well-known class of optimization and the literature term is “late materialization”. It is a large set of strategies including this one. Late materialization is about as o…

The equivalent to this feature is one of my favorite parts of Husky, Datadog’s storage and query system for event data. https://youtu.be/mNneCaZewTg?si=N68fsBlYS3tuvLe3 begins at 3…

We're still drafting our next post in this series, but the answer is actually very simple: two tiers of object storage do not have the same drawbacks as a combination of object sto…

This strategy will not work well for Apache Kafka because it is extremely IOPS hungry if you have more than a few partitions, and a replay of a large topic will require lots of IO …

(WarpStream co-founder here) We're not talking about no disks as in no storage, just nothing other than object storage. This does have a latency trade-off, but with the advent of S…

Co-founder of WarpStream here. We wrote a short article explaining how the economics of this new storage class work for a system like ours: https://www.warpstream.com/blog/s3-expre…

WarpStream co-founder here. Implementing the Idempotent Producer feature for the Apache Kafka protocol was definitely a fun challenge for us. Please let me know if you have any que…

[WarpStream co-founder] WarpStream doesn't implement compacted topics today. It is on our roadmap, though. Compacted topics are typically not used in high-throughput workloads, so …

[WarpStream CTO here] WarpStream flushes after 4MiB of data or a configurable amount of time. Flushes can also happen concurrently. In general, we'd prefer to not introduce many kn…

[WarpStream CTO here] I'm not going to respond to your comment directly (we've already solved all the problems you've mentioned), but I thought I should mention for the sake of the…

We're aiming for per-GB usage-based pricing that is significantly cheaper than the alternatives, but the BYOC model combined with our extremely efficient cloud control plane gives …

[WarpStream co-founder here] Please sign up for our mailing list! We'll have some interesting things to announce related to this soon.

With an appropriately configured client (i.e. one that retries and waits for requests to be acknowledged), another Agent would receive the retry and the event would be written to t…

[WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especial…

[WarpStream co-founder here] My co-founder and I worked at Datadog for over 3 years where we built Husky, an event storage and query system built directly on top of S3 as well. We …

[WarpStream CTO here] WarpStream is Kafka protocol compatible, so we do support topic-partitions and consumer groups. We do not expose support for transactions or idempotent produc…

(WarpStream CTO here) This is a bit subtle I will admit, but we view the Kafka protocol as a successor here because it will outlive Kafka the implementation.

[WarpStream co-founder here] That is correct about flushing. RE: consuming. The TLDR; is that the agents in an availability zone cluster with each other to form a distributed file …