Scaling Kafka at Honeycomb
honeycomb.io
Scaling Kafka at Honeycomb
1–10 of 47 posts
Re: Scaling Kafka at Honeycomb
#2>> Historically, our business requirements have meant keeping a buffer of 24 to 48 hours of data to guard against the risk of a bug in retriever corrupting customer data.
I have used much larger buffers before. Some bugs can lurk around for a while before noticed. For example the lack of something is much harder to notice.
Re: Scaling Kafka at Honeycomb
#3>> Historically, our business requirements have meant keeping a buffer of 24 to 48 hours of data to guard against the risk of a bug in retriever corrupting customer data. I have used much larger buffers before. Some bugs can lurk around for a while before noticed. For example the lack of something is much harder to notice.
Yes -- now that we're just paying the cost to store once on S3 rather than 3x on NVMe, we can plausibly extend the window to 72 hours or longer! before, it was a pragmatic, constraint-driven compromise.
Re: Scaling Kafka at Honeycomb
#4> …RedPanda, a scratch backend rewrite in Rust that is client API compatible
I thought RedPanda was mostly C++?
Re: Scaling Kafka at Honeycomb
#5> …RedPanda, a scratch backend rewrite in Rust that is client API compatible I thought RedPanda was mostly C++?
The RedPanda website claims to be written in C++, and their open source github repo agrees.
Re: Scaling Kafka at Honeycomb
#6> …RedPanda, a scratch backend rewrite in Rust that is client API compatible I thought RedPanda was mostly C++?
The RedPanda website claims to be written in C++, and their open source github repo agrees.
thanks for the correction! knew an error would slip in there somewhere! apparently they have considered rust though! https://news.ycombinator.com/item?id=25112601
it's fixed now.
Re: Scaling Kafka at Honeycomb
#7> July 2019 we did a rolling restart to convert from self-packaged Kafka 0.10.0
Ouch, that's a lot of fixed bugs you weren't reaping the benefits of >_After we hit a few bad ones that finally convinced our sysop team to move past 0.11.x, life was far better - especially recovery speed after an unclean shutdown. Used to take two hours, dropped to like 10 minutes.
There was a particular bug I can't find for the life of me that we hit about four times in one year where the replicas would get confused about where the high watermark was, and refuse to fetch from the leader. Although to be fair to Kafka 0.10.x, I think that was a bug introduced in 0.11.0. Which is where I developed my personal philosophy of "never upgrade to a x.x.0 Kafka release if it can be avoided."
> The toil of handling reassigning partitions during broker replacement by hand every time one of the instances was terminated by AWS began to grate upon us
I see you like Cruise Control in the Confluent Platform, did you try it earlier?
> In October 2020, Confluent announced Confluent Platform 6.0 with Tiered Storage support
Tiered storage is slowly coming to FOSS Kafka, hopefully in 3.2.0, thanks to some very nice developers from AirBnB. Credit to the StreamNative team, that FOSS Pulsar has tiered storage (and schema registry) built-in.
Re: Scaling Kafka at Honeycomb
#8> July 2019 we did a rolling restart to convert from self-packaged Kafka 0.10.0 Ouch, that's a lot of fixed bugs you weren't reaping the benefits of >_ After we hit a few bad ones that finally convinced our sysop team to move past 0.11.x, life was far better - especially recovery speed after an unclean shutdown. Used to take two hours, dropped to like 10 minutes. There was a particular bug I can't find for the life o…
> What was the reason to stick on 0.10.0 for so long?
Aforementioned self-packaging, we were mangling the .tar.gz files into .debs, and we had to remember to update the debs and then push them out onto our systems, instead of just using Apt. Thus why Confluent's prebuilt distro helped a lot! But also the team was just _afraid_ of Kafka and didn't want to touch it unnecessarily.
> I see you like Cruise Control in the Confluent Platform, did you try it earlier?
We definitely should have. We tried Datadog's Kafka-kit but found adapting it to use Wavefront or Honeycomb Metrics products was more problematic than it needed to be.
> Tiered storage is slowly coming to FOSS Kafka, hopefully in 3.2.0, thanks to some very nice developers from AirBnB. Credit to the StreamNative team, that FOSS Pulsar has tiered storage built-in.
Yeah, we're glad the rest of the world gets to have it, and also glad we paid upfront for Confluent's enterprise feature version to get us out of the immediate bind we had in 2020. Those EBS/instance storage bills were adding up fast.
Re: Scaling Kafka at Honeycomb
#9I've never used Kafka but this post is yet another hard earned lesson in log replication systems where storage tiering should be much higher on the hierarchy of needs than horizontal scaling of individual logs/topics/streams. In my experience the times when you need storage tiering something awful is already happening.
During network partitions or other scenarios where your disks are filling up quickly it's much easier to reason about how to get your log healthy by aggressively offloading to tiered storage and trimming than it is to re-partition (read: reconfigure), which often requires writes to some consensus-backed metadata store, which is also likely experiencing its own issues at that time.
Another great benefit of storage tiering is that you can externally communicate a shorter data retention period than you actually have in practice, while you really put your recovery and replay systems through their paces to get the confidence you need. Tiered storage can also be a great place to bootstrap new nodes from.
Re: Scaling Kafka at Honeycomb
#10> July 2019 we did a rolling restart to convert from self-packaged Kafka 0.10.0 Ouch, that's a lot of fixed bugs you weren't reaping the benefits of >_ After we hit a few bad ones that finally convinced our sysop team to move past 0.11.x, life was far better - especially recovery speed after an unclean shutdown. Used to take two hours, dropped to like 10 minutes. There was a particular bug I can't find for the life o…
> What was the reason to stick on 0.10.0 for so long? Aforementioned self-packaging, we were mangling the .tar.gz files into .debs, and we had to remember to update the debs and then push them out onto our systems, instead of just using Apt. Thus why Confluent's prebuilt distro helped a lot! But also the team was just _afraid_ of Kafka and didn't want to touch it unnecessarily. > I see you like Cruise Control in the…
> we paid upfront for Confluent's enterprise feature version to get us out of the immediate bind we had in 2020.
Definitely agree it's an essential feature for large datasets - in the past I've used Kafka Connect to stream data to S3 for longer term retention, but it's something else to manage, and getting data back into a topic if needed can be a bit painful.