171–180 of 249 posts
Earlier quoted context omitted.
We provide a SaaS offering of Apache Pulsar in AWS, Azure, and GCP: https://kafkaesque.io/
Cool name. That's one of those company names that almost seems like someone thought it would make a good company name first and thought it was so fitting, they should build a company around it.
Splunk just acquired streamlio and most of the core devs got sucked up. While pulsar is a great product - are you not concerned that these guys are getting paid $$ bank to do something else now?
Splunk just acquired streamlio and most of the core devs got sucked up. While pulsar is a great product - are you not concerned that these guys are getting paid $$ bank to do something else now?
spoiler: we're still working on Pulsar
Earlier quoted context omitted.
> You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar. Could you elaborate why this would be the case?
Not the OP, but I think they were exaggerating a bit. In practice, operating kafka is a major PITA, because it means you have to (1) choose a "flavor" wrapper (confluent seems to be a popular one), because the base project isn't easy to develop against (2) write your own wrappers of those wrappers, to keep your developers from shooting themselves in the foot with wacky defaults (3) suffer the immense pain that is aut…
How is it compared to kafka?
Most of the flaws of Kafka are carefully studied and fixed in Apache pulsar. I have written a blog about why we went ahead with pulsar https://medium.com/@yuvarajl/why-nutanix-beam-went-ahead-wit...
I just want to clarify this - you're limited to N concurrent consumers for N partitions per consumer group.
Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…
It's an interesting observation. I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems). I will almost certainly get downvoted for this (as I always…
Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit to…
I can't wait for projects to ditch ZooKeeper. Apache Bookkeeper, which Apache Pulsar uses for its state, already supports Etcd as a consensus store (though I believe this is still alpha? beta? quality). Pulsar is also working on supporting Etcd.
Earlier quoted context omitted.
Normally don't plug my own work, but this is super related. Did you ever check out Stream? https://getstream.io/ We power chat and feeds for >500 million end users. Tech is Go, RocksDB & Raft based.
Yes, we did indeed consider Stream, but figured we could save some money by deploying and running our own system. We are very hopeful to quickly get a couple million users in a short time from our launch and that would have ran up our costs with stream quickly.
We've had some really large companies move to Stream from their in-house tech and save 30-70% comparing Stream's monthly fees vs their in-house hosting. (the difference gets much larger if you add the engineering & maintenance cost of their in-house systems). If your team is in the USA & funded it's pretty difficult to build in-house with a good ROI.
Earlier quoted context omitted.
Not questioning your judgement but interested to know about the factors moving you away from Kinesis.
Biggest pain points with Kinesis: - ordering is really hard, you don't get guaranted ordering unless you write one message at a time or do a lot of complexity on writes (see https://brandur.org/kinesis-order ) and the shards are simply too small for many of our ordered use cases - cost, we just don't send some data right now because it would just be too much relative to the utility of the data (we would need like 250…
Do you need to be able to generate a materialized view for a specific time window?
It feels weird to me to use your pub sub system to handle your persistent storage for views, but I am definitely missing context into pulsar and your use case
Earlier quoted context omitted.
Not the OP, but I think they were exaggerating a bit. In practice, operating kafka is a major PITA, because it means you have to (1) choose a "flavor" wrapper (confluent seems to be a popular one), because the base project isn't easy to develop against (2) write your own wrappers of those wrappers, to keep your developers from shooting themselves in the foot with wacky defaults (3) suffer the immense pain that is aut…
What do you mean by 1 and 2? I'm guessing you're referring to the kafka-clients API? The defaults for producer and consumer conf are quite sensible these days.
In this churny environment, where you want to keep on latest versions (necessitated by bugs mentioned in), you need abstractions to protect you somewhat from the churn.
Confluent also seems to have a fair amount of churn, so you need wrappers for that, that you can update all at once for your developers.