Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

91–100 of 295 posts

Re: Kafka is dead, long live Kafka

#91

Post by jkaplowitz »

lxe wrote: post #43
nfw2 wrote: post #38

The title of the article should be "Kafka is dead. Long live Warpstream." The long live part refers to the successor.

You're right! I've only heard it being used as a contradictory phrase though. https://en.wikipedia.org/wiki/The_king_is_dead,_long_live_th... !

As per your link, it's only "seemingly" contradictory - that is, it emphasizes that as soon as the (previous) king dies, the (new) king is automatically immediately in office to be wished a long life, without needing to wait for any coronation ceremony. This is the rule in many monarchies past and present, though not all. So, imagine that the (old) king dies at the end of the first clause and therefore the (new) king takes office at the beginning of the second.

Re: Kafka is dead, long live Kafka

#92

Post by kfk »

carlio wrote: post #78
slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem. I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

The biggest issue for me is people using kafka for mqtt, mqtt is a pub/sub broker already. The other issue is thinking of Kafka as some kind of “innovative” data ingestion tool, so now instead of 50 extract jobs per day, you got to reconcile millions of events in realtime. I think message brokers make sense, but they are message brokers, nothing else, no?

Re: Kafka is dead, long live Kafka

#93

Post by necubi »

richieartoul wrote: post #23
jiaaro wrote: post #8

Earlier quoted context omitted.

The blog post mentions that partitions are too low-level an abstraction to program against. Does that mean WarpStream doesn't use partitions? Do you provide any ordering guarantees like Kafka does at the partition level?

(WarpStream founder) No WarpStream has partitions internally and provides the same ordering guarantees Kafka does at the partition level. We're just saying that we think for most streaming applications this is not a great programming model, and we think there is an opportunity to do something better (but we haven't done that yet).

Do you still provide low-level control over partition subscriptions and offset management? Any plan to support Kafka transactions?

That's all required to build exactly-once systems on top of Kafka (like the stateful stream processing engine I work on) even if it's not the easiest interface for normal application-level development.

Re: Kafka is dead, long live Kafka

#95

Post by nithril »

derefr wrote: post #82
nithril wrote: post #69

Earlier quoted context omitted.

> Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? Azure disk replications are for the durability of the data, not the avalability of the data from a kafka perspective.

Many systems might just be using Kafka to drive async batch transaction processing (think: sending emails, charging credit cards), and therefore don't care at all about availability.

From a read perspective I would agree but from a write perspective if the partition is not available? (genuinely asking)

Re: Kafka is dead, long live Kafka

#97

Post by rubiquity »

derefr wrote: post #81
rubiquity wrote: post #3

Earlier quoted context omitted.

Congrats! "The SQLite of Kafka" is an item from my side projects pile I'm happy to delete. One reason I never built it is because it felt paradoxical that users might want a scaled down Kafka rather than using SQLite directly if the scale didn't matter. But you may find out that people enjoy the semantics of the Kafka protocol or are already using Kafka and have learned they don't have the scale they thought they did…

> it felt paradoxical that users might want a scaled down Kafka rather than using SQLite directly if the scale didn't matter. I don't need to push very many messages (not enough to justify running Kafka), but each of the messages that I do push are both 1. very important and must be cross-AZ durable, and 2. very urgent and must not be blocked by e.g. contended writes in a regular RDBMS. Currently, the winner of this…

I can't speak to GCP or Azure but the semantics of a log offer replayability whereas SQS does not.

Re: Kafka is dead, long live Kafka

#98

Post by dopidopHN »

ram_rar wrote: post #77
slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?

RabbitMQ, ActiveMQ, MQtt

All of those are fine is you only need pub/sub

Re: Kafka is dead, long live Kafka

#99

Post by nithril »

carlio wrote: post #78
slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem. I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

Same finding.

That's why that sentence in the article "but almost every technology company uses it." should be rephrased to "but almost every technology company do not need it"

Re: Kafka is dead, long live Kafka

#100

Post by richieartoul »

pdeva1 wrote: post #90
richieartoul wrote: post #35

Earlier quoted context omitted.

(WarpStream founder) 1. Yeah, we mention at the end of the post the P99 produce latency is ~400ms. 2. MSK still charges you for networking to produce into the cluster and consumer out of it if follower fetch is not properly configured. Also, you still have to more or less manage a Kafka cluster (hot spotting, partition rebalancing, etc). In practice we think WarpStream will be much cheaper to use than MSK for almost…

1. what payload size and flush interval is that latency measured against?

1. By payload size do you mean record size? They're ~1KiB 2. Flush interval was 100ms, the agent defaults to 250ms though I believe.
Post reply on HN