Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Kafka at Honeycomb

honeycomb.io

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Scaling Kafka at Honeycomb

#31

Post by krnaveen14 »

staticassertion wrote: post #30
krnaveen14 wrote: post #27

Based on our experience with Apache Kafka and alternative streaming systems, Apache Pulsar natively addresses the Honeycomb's needs. - Decoupling of Broker & Storage Layer - Tierered Storage (SSD, HDD, S3,...) We use both Kafka and Pulsar in our systems. - Kafka is used for microservices communication and operational data sharing - Pulsar is used for streaming large customer data in thousands of topics

What's the story like for migrating from Kafka to Pulsar?

- Kafka couldn't cope up if there are hundreds or few thousands topics. High CPU load, longer startup times...

- Even empty Kafka topic consumes 20MB of on-disk storage (that's 20GB for 1000 topics)

- Inevitable coupling of non-partitioned topic to a particular Kafka broker limiting the storage scale

- Tiered storage was not available previously in Kafka (is it now available in open source version?)

- Native Multi Tenant support with Authentication / Authorization support not available in Kafka (essential for customer based thousands of namespaces and topics)

Re: Scaling Kafka at Honeycomb

#32

Post by helsinki »

krnaveen14 wrote: post #31
staticassertion wrote: post #30

Earlier quoted context omitted.

What's the story like for migrating from Kafka to Pulsar?

- Kafka couldn't cope up if there are hundreds or few thousands topics. High CPU load, longer startup times... - Even empty Kafka topic consumes 20MB of on-disk storage (that's 20GB for 1000 topics) - Inevitable coupling of non-partitioned topic to a particular Kafka broker limiting the storage scale - Tiered storage was not available previously in Kafka (is it now available in open source version?) - Native Multi Te…

> Kafka couldn't cope up if there are hundreds or few thousands topics. High CPU load, longer startup times...

It also uses num_partitions*2 open file descriptors per topic, which can quickly surpass the default ulimit on a host. Always remember to raise the ulimit before you near 1000 topics, otherwise Kafka crashes.

Re: Scaling Kafka at Honeycomb

#33

Post by rad_gruchalski »

lizthegrey wrote: post #11
EdwardDiego wrote: post #10

Earlier quoted context omitted.

> we paid upfront for Confluent's enterprise feature version to get us out of the immediate bind we had in 2020. Definitely agree it's an essential feature for large datasets - in the past I've used Kafka Connect to stream data to S3 for longer term retention, but it's something else to manage, and getting data back into a topic if needed can be a bit painful.

getting to just use the same consistent API without rewriting clients was AMAZING.

This always reminds me of this thing I’ve written some time ago : https://gruchalski.com/posts/2021-04-02-kafka-infinite-reten...

Re: Scaling Kafka at Honeycomb

#34

Post by bashtoni »

This is an awesome write up. I love reading these warts and all accounts - they're always way more useful than the typical case study "we switched to X and it saved us Y%!" marketing posts.

One point that makes Intel not look quite so bad performance wise - based on my own benchmarking, I'm pretty sure when this article talks about cores they actually mean vCPUs. In AWS on x86, 1 vCPU is 1 hyperthread, so it's kind of half a core. On Graviton 2, 1 vCPU is one full core, the CPUs don't have hyperthreading. This means that you need 10 Intel cores to do the same work as 16 Graviton cores, not 20. This of course doesn't change the cost savings from switching to arm64.

Re: Scaling Kafka at Honeycomb

#35

Post by camel_gopher »

lizthegrey wrote: post #18
rmb938 wrote: post #15

Maybe I missed it but are you able to talk about how many messages a second, partition count and average message size? I run a few hundred Kafka clusters with message counts per second in the tens of millions for some clusters, a few thousand partitions, message sizes around 7kb with gzip compression, and have never needed the amount of CPU and network/disk throughput mentioned. With node counts range between ~10-25.…

1.5M messages/sec, average message size 1kb pre compression, 300 bytes post compression/batching. the problem was that we were really really disk limited before for keeping the 48 hour window of data, having to keep everything on NVMe or EBS was astoundingly expensive. but yeah, we run it all off 6 brokers now.

Did you look at any of the other solutions such as fq? 300 bytes is a solidly small size. I’m guessing Kafka has gotten faster since this doc was published, but might be worth investigating. https://github.com/neophenix/StateOfTheMQ/blob/master/state_...

Re: Scaling Kafka at Honeycomb

#37

Post by lizthegrey »

krnaveen14 wrote: post #31
staticassertion wrote: post #30

Earlier quoted context omitted.

What's the story like for migrating from Kafka to Pulsar?

- Kafka couldn't cope up if there are hundreds or few thousands topics. High CPU load, longer startup times... - Even empty Kafka topic consumes 20MB of on-disk storage (that's 20GB for 1000 topics) - Inevitable coupling of non-partitioned topic to a particular Kafka broker limiting the storage scale - Tiered storage was not available previously in Kafka (is it now available in open source version?) - Native Multi Te…

right -- the question I and others have is, can you convert an existing topic in place or do you have to do the dual-writer/shadow launch, then drain the old one model etc.

Re: Scaling Kafka at Honeycomb

#38

Post by lizthegrey »

bashtoni wrote: post #34

This is an awesome write up. I love reading these warts and all accounts - they're always way more useful than the typical case study "we switched to X and it saved us Y%!" marketing posts. One point that makes Intel not look quite so bad performance wise - based on my own benchmarking, I'm pretty sure when this article talks about cores they actually mean vCPUs. In AWS on x86, 1 vCPU is 1 hyperthread, so it's kind o…

Intel looks less bad when you compare one physical core to one physical core, but AWS definitely will not sell you one physical Intel core for the price of one ARM core, they will sell you _half_ of one Intel core for 20% more than an ARM core.

Re: Scaling Kafka at Honeycomb

#39

Post by lizthegrey »

camel_gopher wrote: post #35
lizthegrey wrote: post #18

Earlier quoted context omitted.

1.5M messages/sec, average message size 1kb pre compression, 300 bytes post compression/batching. the problem was that we were really really disk limited before for keeping the 48 hour window of data, having to keep everything on NVMe or EBS was astoundingly expensive. but yeah, we run it all off 6 brokers now.

Did you look at any of the other solutions such as fq? 300 bytes is a solidly small size. I’m guessing Kafka has gotten faster since this doc was published, but might be worth investigating. https://github.com/neophenix/StateOfTheMQ/blob/master/state_...

Ah, https://github.com/circonus-labs/fq

It was less mature in 2016 when we made the original technology choice (and is still, I'd say, probably not a Boring Technology today). With batching, Kafka is plenty fast for us!

Re: Scaling Kafka at Honeycomb

#40

Post by polskibus »

krnaveen14 wrote: post #27

Based on our experience with Apache Kafka and alternative streaming systems, Apache Pulsar natively addresses the Honeycomb's needs. - Decoupling of Broker & Storage Layer - Tierered Storage (SSD, HDD, S3,...) We use both Kafka and Pulsar in our systems. - Kafka is used for microservices communication and operational data sharing - Pulsar is used for streaming large customer data in thousands of topics

Is Pulsar a good fit for microservices communication ?
Post reply on HN