Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Kafka at Honeycomb

honeycomb.io

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Scaling Kafka at Honeycomb

#21

Post by rmb938 »

lizthegrey wrote: post #20
throwdbaaway wrote: post #19

Earlier quoted context omitted.

If I understand correctly, there were: - issues with tail latency and cost when using gp2 - issues with generally bad performance when using st1 - issues with reliability when using gp3 (as an early adopter of aws "GA" product) - issues with insufficient disk space when using local-attached nvme - issues with confluent licensing cost And tiered storage solves all of that. The thing is, I have not seen kafka strugglin…

Oh, believe me, we have hired Corey Quinn (Duckbill Group). AWS budged on some things, but not on the EBS cost.

Maybe it's worth trying out GCP for a POC cluster? Downside is they don't have any ARM instances but some back of the napkin math does show that an equivalent setup of using 6x im4gn.2xlarge in AWS to 6xn2-standard-8 and 6x3750gb pd-balanced ssds is roughly around the same cost and disk perf, could be a bit cheaper with their AMD instances instead of Intel. If you compare it to a gp2 disk it's roughly 4x faster, but the same per for local disk on im4gn.2xlarge.

I also have had decent success with getting committed use and reservations on GCP ssd compared to other cloud providers.

Re: Scaling Kafka at Honeycomb

#22

Post by lizthegrey »

rmb938 wrote: post #21
lizthegrey wrote: post #20

Earlier quoted context omitted.

Oh, believe me, we have hired Corey Quinn (Duckbill Group). AWS budged on some things, but not on the EBS cost.

Maybe it's worth trying out GCP for a POC cluster? Downside is they don't have any ARM instances but some back of the napkin math does show that an equivalent setup of using 6x im4gn.2xlarge in AWS to 6xn2-standard-8 and 6x3750gb pd-balanced ssds is roughly around the same cost and disk perf, could be a bit cheaper with their AMD instances instead of Intel. If you compare it to a gp2 disk it's roughly 4x faster, but…

It would be for GCP based customers, but we are a telemetry platform and making our (majority) AWS based customers pay 0.08 per GB to egress to us is a non-starter for them :/

Re: Scaling Kafka at Honeycomb

#23

Post by lizthegrey »

mherdeg wrote: post #12

It's funny how my bugbears from interacting with distributed async messaging (Kafka) are like 90 degrees orthogonal from the things described here: (1) Occasionally have wanted to wonder what the actual traffic is . This takes extra software work (writing some kind of inspector tool to consume a sample message and produce a human-readable version of what's inside it). (2) Sometimes see problems which happen at the br…

Yup, we went with fewer, higher throughput partitions for this exact reason of wanting precise control over partitioning. It's been a blessing and a curse.

Re: Scaling Kafka at Honeycomb

#24

Post by ryan_lane »

ewhauser421 wrote: post #13
lizthegrey wrote: post #8

Earlier quoted context omitted.

> What was the reason to stick on 0.10.0 for so long? Aforementioned self-packaging, we were mangling the .tar.gz files into .debs, and we had to remember to update the debs and then push them out onto our systems, instead of just using Apt. Thus why Confluent's prebuilt distro helped a lot! But also the team was just _afraid_ of Kafka and didn't want to touch it unnecessarily. > I see you like Cruise Control in the…

The irony of Honeycomb using an open source tool from Datadog is not lost on me =)

This is one of the major benefits of open source. You can share engineering resources, even with your competitors, for things that aren't your bread & butter.

Re: Scaling Kafka at Honeycomb

#25

Post by lizthegrey »

ryan_lane wrote: post #24
ewhauser421 wrote: post #13

Earlier quoted context omitted.

The irony of Honeycomb using an open source tool from Datadog is not lost on me =)

This is one of the major benefits of open source. You can share engineering resources, even with your competitors, for things that aren't your bread & butter.

Yup! In fact, we collaborate with Datadog, New Relic, Splunk, Lightstep, et al on OpenTelemetry (https://news.ycombinator.com/item?id=28997275 / opentelemetry.io)

Re: Scaling Kafka at Honeycomb

#26

Post by juliansimioni »

I'm not a Kafka user but it seems very similar to my experience running Elasticsearch clusters tuned for low latency response time.

There's a complicated mix of requirements for CPU, memory, disk, _and_ network speed, and meeting all of them cost effectively is a real challenge.

Similarly, it's easy to build a cluster that performs well until a single node fails. The increased load per node plus the CPU and network cost of replicating data to a replacement instance can really cause trouble.

Elasticsearch also runs on the JVM so I'm hoping the new EC2 instance types will work for us too. They look to be really great.

Re: Scaling Kafka at Honeycomb

#27

Post by krnaveen14 »

Based on our experience with Apache Kafka and alternative streaming systems, Apache Pulsar natively addresses the Honeycomb's needs.

- Decoupling of Broker & Storage Layer

- Tierered Storage (SSD, HDD, S3,...)

We use both Kafka and Pulsar in our systems.

- Kafka is used for microservices communication and operational data sharing

- Pulsar is used for streaming large customer data in thousands of topics

Re: Scaling Kafka at Honeycomb

#28

Post by throwaway984393 »

I'm still just reading the first couple sections, but I already want to give props for an excellent write-up. You're explaining in (mostly) plain language the purpose of Kafka, your specific application of it, and lots of great background detail and history of your implementation's evolution. It's also clear that whoever wrote this knows their Ops. Thank you!

Re: Scaling Kafka at Honeycomb

#29

Post by lizthegrey »

krnaveen14 wrote: post #27

Based on our experience with Apache Kafka and alternative streaming systems, Apache Pulsar natively addresses the Honeycomb's needs. - Decoupling of Broker & Storage Layer - Tierered Storage (SSD, HDD, S3,...) We use both Kafka and Pulsar in our systems. - Kafka is used for microservices communication and operational data sharing - Pulsar is used for streaming large customer data in thousands of topics

In 2016 when we were founded, Pulsar wasn't a thing yet. Today, we'd be very likely to use Pulsar if we were starting from scratch.

Re: Scaling Kafka at Honeycomb

#30

Post by staticassertion »

krnaveen14 wrote: post #27

Based on our experience with Apache Kafka and alternative streaming systems, Apache Pulsar natively addresses the Honeycomb's needs. - Decoupling of Broker & Storage Layer - Tierered Storage (SSD, HDD, S3,...) We use both Kafka and Pulsar in our systems. - Kafka is used for microservices communication and operational data sharing - Pulsar is used for streaming large customer data in thousands of topics

What's the story like for migrating from Kafka to Pulsar?
Post reply on HN