Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

141–150 of 295 posts

Re: Kafka is dead, long live Kafka

#141

Post by richieartoul »

guywithabike wrote: post #105
nathas wrote: post #94

Is WarpStream considering hiring Aphyr to do a Jepsen test?

WarpStream relies on a proprietary metadata store hosted within their internal network to operate, so it's pretty unlikely that Jepsen tests could cover that. If you're ok with the externally hosted metadata stores as well as the high per-request latencies (p99 of 400ms, according to WarpStream), it's highly likely that things like liveness and safety properties are pretty far from your mind. So, I wouldn't bank on t…

(WarpStream co-founder)

FWIW we subject WarpStream to continuous chaos/fault injection in our integration tests and staging environment to verify correctness and liveness properties. I wouldn't say they're far from our mind, we've just made a big trade off around latency that we think will make sense for a lot of people.

Re: Kafka is dead, long live Kafka

#142

Post by pshirshov »

richieartoul wrote: post #45
pshirshov wrote: post #39

Ok, intsead of providing commit log guarantees on their own they rely on S3. I'm not sure if S3 can actually provide durable commit log guarantees. There is timestamp-based last write wins policy for concurrent writes, so I'm not sure if this thingy can actually replace Kafka in all the usecases providing same guarantees.

(WarpStream co-founder) We use S3 for data replication, availability, and durability, but not for the log ordering guarantees. For that, we have our own custom metadata store that "imposes" ordering guarantees on top of the data that is written to S3. We don't rely on any kind of timestamp-based last write wins policy for concurrent writes. In fact the agents will never write a file to S3 with the same name more than…

So, in fact it's not really stateless, you have your locking/ordering layer and you implement some consensus for that layer?

Re: Kafka is dead, long live Kafka

#143

Post by mdaniel »

benjaminwootton wrote: post #119
slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

I’m not sure how anyone can hate Kafka? It does what it says on the tin - move data from A to B with publish/subscribe semantics. It’s quite easy to just use it as a dumb message broker with no retention if that’s all you need but if you do want to do something funky with persistence then go down that route. I’m not sure how anyone could have a negative feeling towards a vanilla, but rock solid and wildly popular ope…

> I’m not sure how anyone can hate Kafka?

My experience has been that such a question has two implied audiences in it: those who consume Kafka and those who have to keep the PoS alive and healthy

The whole ambiguity around whether ZK is really still needed or not https://kafka.apache.org/documentation/#zk> makes keeping two distributed systems alive and healthy, but don't worry you can't move your production clusters off of it anyway https://kafka.apache.org/documentation/#kraft_zk_migration>. It's a mess

Re: Kafka is dead, long live Kafka

#144

Post by mdaniel »

yingjunwu wrote: post #126

I am the founder of RisingWave ( http://risingwave.com/ ), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion: * Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka proto…

If you're still within the edit window for your comment, the ");" in them are attached to the URLs, making them problematic to click

ed

-- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/

-- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...

-- Redpanda Data, which cuts your TCO by 6X https://redpanda.com/platform-tco

-- DataStax, which commercializes Apache Pulsar https://pulsar.apache.org/

-- StreamNative, which commercializes Apache Pulsar https://pulsar.apache.org/

Re: Kafka is dead, long live Kafka

#145

Post by deepsquirrelnet »

snapetom wrote: post #106
carlio wrote: post #78

Earlier quoted context omitted.

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem. I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

> and every time it has been a solution in search of a problem. More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

It’s amazing how far that one word can go.

In a former life, I even heard this one:

- We use CentOS.

Why?

- RedHat is enterprise.

(But you’re not even paying for enterprise support)

Re: Kafka is dead, long live Kafka

#146

Post by parhamn »

benjaminwootton wrote: post #128
parhamn wrote: post #88

Earlier quoted context omitted.

Kafka is a closer to a persistent WAL than a message queue. If your work doesn't need a WAL, it's almost certainly overkill and you will hate it. If your work needs a WAL then it'll be your favorite tool ever.

Why? Its quite easy to use Kafka as a messaging queue without even thinking about the write ahead log semantics. It’s there if you need it, but Kafka scales down to being a message broker fairly well in my opinion.

Because operationalizing Kafka is difficult from a infrastructure (scala/java, zookeper, durable disk management, lots of moving parts), learning and a code perspective (pointer tracking, partition management, delegation, etc) relative to the other pubsub/mq tools.

So if you don't have it operationalized and your use case is simple, it makes most sense to use a simpler tool (rmq/ampq, cloud pubsub, nsq, etc, perhaps even redis)

Re: Kafka is dead, long live Kafka

#147

Post by rad_gruchalski »

Sphax wrote: post #79
rad_gruchalski wrote: post #50

I’m so happy to see that someone has finally built such thing. I’ve been asking for tiered storage in Kafka since 2016 ( https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-c... ) but this approach is so much better. Kafka itself could solve this with tiered storage OR, at least, allowing adding volumes at runtime. But neither is possible. Very cool, have to find some time to take this for a test drive.

Just fyi, tiered storage is coming at some point in Kafka: https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...

It’s been like that for a while.

Re: Kafka is dead, long live Kafka

#148

Post by insanitybit »

saryant wrote: post #80
insanitybit wrote: post #62

Earlier quoted context omitted.

Getting a distributed set up plugged into your k8s/whatever framework can be a pain.

The issue there is running it on k8s. On bare metal or even VMs it’s not at all hard to run even at scale.

Well when all of your services are in k8s, and your service discovery is on k8s, etc, you tend to put things like Kafka into K8s.

Re: Kafka is dead, long live Kafka

#149

Post by rad_gruchalski »

richieartoul wrote: post #56
rad_gruchalski wrote: post #50

I’m so happy to see that someone has finally built such thing. I’ve been asking for tiered storage in Kafka since 2016 ( https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-c... ) but this approach is so much better. Kafka itself could solve this with tiered storage OR, at least, allowing adding volumes at runtime. But neither is possible. Very cool, have to find some time to take this for a test drive.

(WarpStream co-founder) Thanks for the kind words! I'll just add that tiered storage is not quite the same thing because it means you still have to manage local disks and replication carefully, however briefly. Please reach out with questions any time!

Indeed. The way I was thinking about it was to implement a different file system for it.

Re: Kafka is dead, long live Kafka

#150

Post by EdwardDiego »

carlio wrote: post #78
slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem. I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

It's really good when you're producing a lot of data fast that you don't want to lose, and you want multiple consumers to read.

It's a complex tool that solves a complicated problem. But if you don't actually have that problem, then that's a whole lot of complexity for no gain.

Post reply on HN