Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

111–120 of 295 posts

Re: Kafka is dead, long live Kafka

#111
post #93

Earlier quoted context omitted.

(WarpStream founder) No WarpStream has partitions internally and provides the same ordering guarantees Kafka does at the partition level. We're just saying that we think for most streaming applications this is not a great programming model, and we think there is an opportunity to do something better (but we haven't done that yet).

Do you still provide low-level control over partition subscriptions and offset management? Any plan to support Kafka transactions? That's all required to build exactly-once systems on top of Kafka (like the stateful stream processing engine I work on) even if it's not the easiest interface for normal application-level development.

[WarpStream CTO here]

WarpStream is Kafka protocol compatible, so we do support topic-partitions and consumer groups. We do not expose support for transactions or idempotent producing today, but the internals of the system support that and we will probably work on the idempotent producer sometime in the next month, with transactions coming shortly after, depending on demand from the Developer Preview users.

Re: Kafka is dead, long live Kafka

#114
post #77

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?

Most of my designs include a message queue, and I would not use Kafka unless there was a strong need.

Right now my preference varies a bit depending on the rest of the tech stack, but for the most part I use Redis or RabbitMQ.

If the stack is already hard dependent on AWS or another cloud, then SQS or whatever is also fine.

I also wouldn't overlook just using your existing DB (like postgres)! At low and even medium scale this can be totally fine, and also comes with lots of benefits like single-source-of-truth, normal relational DB constraints, transaction wrapping, and more. One of the highest scale apps I've worked on uses Postgres for queueing. It's take a number of optimizations over the years as performance starts to fall due to scale, but it's doable.

Re: Kafka is dead, long live Kafka

#115
post #32

Earlier quoted context omitted.

1. dont producers now have much higher latency since they have to wait for writes to s3. 2. if the '5-10x cheaper' is mostly due to cross AZ savings, isnt that offered by AWS MSK offering too?

(WarpStream founder) 1. Yeah, we mention at the end of the post the P99 produce latency is ~400ms. 2. MSK still charges you for networking to produce into the cluster and consumer out of it if follower fetch is not properly configured. Also, you still have to more or less manage a Kafka cluster (hot spotting, partition rebalancing, etc). In practice we think WarpStream will be much cheaper to use than MSK for almost…

How does the cost compare if follower fetch is properly configured?

Re: Kafka is dead, long live Kafka

#116

Oh man, I can't wait until these people get their first S3 bill for the number or GET requests.

[WarpStream co-founder here]

My co-founder and I worked at Datadog for over 3 years where we built Husky, an event storage and query system built directly on top of S3 as well. We know what we're doing here, I promise ;)

https://www.datadoghq.com/blog/engineering/introducing-husky...

Re: Kafka is dead, long live Kafka

#118

Oh man, I can't wait until these people get their first S3 bill for the number or GET requests.

[WarpStream co-founder here] My co-founder and I worked at Datadog for over 3 years where we built Husky, an event storage and query system built directly on top of S3 as well. We know what we're doing here, I promise ;) https://www.datadoghq.com/blog/engineering/introducing-husky...

What happens if I pull the plug on a server that hasn't flushed to S3 in the configured time? Do I loose that event?

Re: Kafka is dead, long live Kafka

#119

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

I’m not sure how anyone can hate Kafka? It does what it says on the tin - move data from A to B with publish/subscribe semantics.

It’s quite easy to just use it as a dumb message broker with no retention if that’s all you need but if you do want to do something funky with persistence then go down that route.

I’m not sure how anyone could have a negative feeling towards a vanilla, but rock solid and wildly popular open source tool. If they do then it will be about some niche feature or use case.

I actually think this message reflects badly on the vendor here. Criticise or compete with Kafka on its technical merits if you like, but this is just a misrepresentation of their position in the market.

Re: Kafka is dead, long live Kafka

#120
post #95
post #82

Earlier quoted context omitted.

Many systems might just be using Kafka to drive async batch transaction processing (think: sending emails, charging credit cards), and therefore don't care at all about availability.

From a read perspective I would agree but from a write perspective if the partition is not available? (genuinely asking)

Think Unix pipelines: writes that can't immediately complete block the producer. (Probably with a bit of an in-memory buffer, but still.) Pile up enough un-ACKed messages to push, and the whole producer should stop consuming its own input end until the consumer on its output end comes back to start pumping messages again. The whole pipeline from the blocked stage back to the start receives backpressure and temporarily stalls out — which is fine, because, again, async batch processing.

And yes, this means that you need to have logic all the way back at the original sender (the one triggering the async message-send as part of some synchronous business-logic), to be able to refuse / abort / revert the entire high-level business-logic operation if the async-message-send's message-accept fails. (A user shouldn't be considered signed up if you can't remember to send them a verification email; a subscription should not be created if you can't remember to charge the card; etc.)

In est, you can think of this as "semi-async": each stage is doing a synchronous RPC call to an "accept and buffer this batch of async messages" endpoint on the broker — which might synchronously fail (if the broker is unavailable, or if the consumer of a bounded-size(!) queue has blocked to create backpressure and therefore the queue has filled and the broker has in turn stopped accepting to that queue.)

With such an API, rather than pretending that there's some magic reliable-delivery system you can "fire and forget" messages onto, these failures gets bubbled up to the caller on the send side, like any other failure of a synchronous RPC call.

Take this to its fullest extent, and you get Google's "effectively synchronous" RPC philosophy, where you have event brokers for routing and discoverability (think k8s Services), but async messages are always either queued in either the sender process's [bounded] outbox, or the recipient process's [bounded] inbox, with no need for a broker-side queue, because everything is designed with backpressure + graceful handling of potential accept failure in mind, including the initial clients knowing to retry pushing the initial message-send. (If you're familiar with the delivery semantics of Golang channels — it's basically that, but distributed rather than process-internal. There's a reason that particular language feature came out of a language designed at Google.)

---

Mind you, there's also the "truly async" batch-processing semantics — the kind ATMs have, where if even the initial client doing a synchronous operation (think: withdrawing cash) can't get in contact with the server/broker to push the async message-sends, then you just append the message to a big ol' local log file, and proceed as if the async sends already succeeded; and then later, when you come back online, you dump your whole built-up log of messages to the broker, and all events in the log are inherently accepted — but there are higher-level semantics that might generate additional revert events in response to some of them (i.e. if the ATM user overdrew their account), that get backfed into the system. But you, as the initial producer of messages, don't have to worry about collating those against your messages or anything like that.

Post reply on HN