Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka at the low end: how bad can it get?

broot.ca

Post reply on HN
Print view View original

131–140 of 144 posts

Re: Kafka at the low end: how bad can it get?

#131

Post by enether »

rockwotj wrote: post #125
munksbeer wrote: post #121

Earlier quoted context omitted.

In kafka, if you require the highest durability for messages, you configure multiple nodes on different hosts, and probably data centres, and you require acks=all. I'd say this is the thing that pushes latency up, rather than the code execution of kafka itself. How does redpanda compare under those constraints?

Oh if you care about durability on Kafka vs Redpanda, see https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s... , acks=all does not fsync (by default before acknowledging the write), so it's still not safe. We use raft for the data path, a proven replication protocol (not the custom ISR protocol) and fsync by default for safety (although if you're good with relaxed durability like in Kafka you can enable th…

It's pretty safe. Kafka replicates to 3 nodes (no fsync) before the request is completed. What are the odds of all 3 nodes (running in different data centers) failing at the same time?

Re: Kafka at the low end: how bad can it get?

#132

Post by enether »

xyst wrote: post #11

> Each of these Web workers puts those 4 records onto 4 of the topic’s partitions in a round-robin fashion. And, because they do not coordinate this, they might choose the same 4 partitions, which happen to all land on a single consumer Then choose a different partitioning strategy. Often key based partitioning can solve this issue. Worst case scenario, you use a custom partitioning strategy. Additionally , why can’t…

Even if you used a 100% random strategy - what OP described can still happen.

Matching the number of consumers can still produce an uneven result too, and OP clarifies that even if the worst case he laid out doesn't happen - in practice there still are idle workers. For the same reason, he likely doesn't want to have 16 workers at all times

Re: Kafka at the low end: how bad can it get?

#133

Post by rockwotj »

enether wrote: post #131
rockwotj wrote: post #125

Earlier quoted context omitted.

Oh if you care about durability on Kafka vs Redpanda, see https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s... , acks=all does not fsync (by default before acknowledging the write), so it's still not safe. We use raft for the data path, a proven replication protocol (not the custom ISR protocol) and fsync by default for safety (although if you're good with relaxed durability like in Kafka you can enable th…

It's pretty safe. Kafka replicates to 3 nodes (no fsync) before the request is completed. What are the odds of all 3 nodes (running in different data centers) failing at the same time?

Generally if you care about safety then “pretty safe” doesn’t cut it.

Re: Kafka at the low end: how bad can it get?

#134

Post by enether »

rockwotj wrote: post #133
enether wrote: post #131

Earlier quoted context omitted.

It's pretty safe. Kafka replicates to 3 nodes (no fsync) before the request is completed. What are the odds of all 3 nodes (running in different data centers) failing at the same time?

Generally if you care about safety then “pretty safe” doesn’t cut it.

It's just my polite way of saying it's safe enough for most use cases and that you're wrong.

The fsync thing is complete FUD by RedPanda. They later introduce write caching[1] and call it an innovation[2]. I notice you also work for them.

Nevertheless, those that are super concerned with safety usually run with an RF of 5 (e.g banks). And you can configure Kafka to fsync as often as you want[3]

1 - https://www.redpanda.com/blog/write-caching-performance-benc... 2 - https://www.linkedin.com/posts/timfox_oh-you-cant-make-this-... 3 - https://kafka.apache.org/documentation/#brokerconfigs_log.fl...

Re: Kafka at the low end: how bad can it get?

#135

Post by enether »

PhilippGille wrote: post #88
enether wrote: post #2

thankfully early access for KIP-932 is coming in 1-3 weeks as the 4.0.0 release gets published

TFA mentions it in the third paragraph: > Note: when Queues for Kafka (KIP-932) becomes a thing, a lot of these concerns go away. I look forward to it!

yep, I was mostly clarifying the timeline

Re: Kafka at the low end: how bad can it get?

#136

Post by agallego »

enether wrote: post #134
rockwotj wrote: post #133

Earlier quoted context omitted.

Generally if you care about safety then “pretty safe” doesn’t cut it.

It's just my polite way of saying it's safe enough for most use cases and that you're wrong. The fsync thing is complete FUD by RedPanda. They later introduce write caching[1] and call it an innovation[2]. I notice you also work for them. Nevertheless, those that are super concerned with safety usually run with an RF of 5 (e.g banks). And you can configure Kafka to fsync as often as you want[3] 1 - https://www.redpan…

It is not FUD. It is deterministic. Reproducible on your laptop. Out of all the banks I work with only a handful of use cases use rf=5. Defaults matter, because most people do not change them.

Re: Kafka at the low end: how bad can it get?

#137

Post by matteomerli »

akshayshah wrote: post #106
gnfargbl wrote: post #101

Earlier quoted context omitted.

The advantages of Pulsar are very much practical, at least for us. Without it we would have to manage two separate messaging systems. I don't see any evidence of StreamNative abandoning Pulsar at this point. I do see a compatibility layer for the Kafka protocol. That's fine.

It sure looks like they’re going quite a ways beyond Kafka-on-Pulsar - the Ursa/Oxia work they’re focused on right now replaces BookKeeper and seems very firmly Kafka-oriented. Or does Ursa also work with the Pulsar protocol?

Ursa supports Pulsar protocol as well

Re: Kafka at the low end: how bad can it get?

#138

Post by deniscoady »

enether wrote: post #134
rockwotj wrote: post #133

Earlier quoted context omitted.

Generally if you care about safety then “pretty safe” doesn’t cut it.

It's just my polite way of saying it's safe enough for most use cases and that you're wrong. The fsync thing is complete FUD by RedPanda. They later introduce write caching[1] and call it an innovation[2]. I notice you also work for them. Nevertheless, those that are super concerned with safety usually run with an RF of 5 (e.g banks). And you can configure Kafka to fsync as often as you want[3] 1 - https://www.redpan…

Disclaimer: I currently work for Redpanda.

It's just my polite way of saying it's safe enough for most use cases and that you're wrong.

Low volume data can be some of the most valuable data on the planet. Think SEC reporting (EDGAR), law changes (Federal Register), court judgements (PACER), new cybersecurity vulnerabilities (CVEs), etc. Missing one record can be detrimental if its the one record that matters.

Does everyone need durability by default? Probably not, but Redpanda users get it for free because there is a product philosophy of default-safe behavior that aligns with user expectations - most folks don't even know how this stuff works, why not protect them when possible?

The fsync thing is complete FUD by RedPanda.

You want durability? Pay the `fsync()` cost. Otherwise recognize that acknowledgement and durability are decoupled and that the data is sitting in unsafe volatile memory for a bit.

They later introduce write caching[1] and call it an innovation[2].

There are legitimate cases where customers don't care about durability and want the fastest possible system. We heard from these folks and responded with a feature they can selectively opt-in for that behavior _knowing the risks_. Again the idea is to be safer by default, and allow folks to opt-in to more risky behaviors.

those that are super concerned with safety usually run with an RF of 5 (e.g banks)

Going above RF=3 does not guarantee "more nines" since you need more independent server racks, independent power supplies or UPSs, etc, otherwise you're just pigeonholing yourself. This greatly drives up costs. Disks and durability is just cheaper and simpler. Worst case you pull the drives and pull the data off them, not fun and not easy, but possible unlike in-memory copies.

And you can configure Kafka to fsync as often as you want[3]

Absolutely! But nobody changes the default which is the issue - expectations of new users are not aligned with actual behavior. Same thing happened during the early MongoDB days. Either there needs to be better documentation/education to have people understand what the durability guarantees actually are, or change the defaults.

Re: Kafka at the low end: how bad can it get?

#139

Post by enether »

VenturingVole wrote: post #113
gnfargbl wrote: post #79

Earlier quoted context omitted.

Pulsar. Works extremely well as both a job queue and a data bus. We have been using it in this application for half a decade now with no serious issues. I don't understand why it doesn't get more popular attention.

Pulsar vs Kafka was a significant lesson to me: The "best" technology isn't always the winner. I put it in quotes because I'm a massive fan of Pulsar and addressing the shortcomings of Kafka. However, with regards to some choices at a former workplace: The broader existing support/integration ecosystem along with Confluent's commercial capabilities won out with regards to technology choices and I was forced to acquie…

and it then becomes a self-perpetuating cycle because the network effect and usage of one product makes it become exponentially more robust & mature than the other. I've heard that Pulsar is buggy basically, and I wouldn't be surprised because it doesn't have thousands of organizations using it in all sorts of ways. It's not its fault too much really

Re: Kafka at the low end: how bad can it get?

#140

Post by enether »

deniscoady wrote: post #138
enether wrote: post #134

Earlier quoted context omitted.

It's just my polite way of saying it's safe enough for most use cases and that you're wrong. The fsync thing is complete FUD by RedPanda. They later introduce write caching[1] and call it an innovation[2]. I notice you also work for them. Nevertheless, those that are super concerned with safety usually run with an RF of 5 (e.g banks). And you can configure Kafka to fsync as often as you want[3] 1 - https://www.redpan…

Disclaimer: I currently work for Redpanda. It's just my polite way of saying it's safe enough for most use cases and that you're wrong. Low volume data can be some of the most valuable data on the planet. Think SEC reporting (EDGAR), law changes (Federal Register), court judgements (PACER), new cybersecurity vulnerabilities (CVEs), etc. Missing one record can be detrimental if its the one record that matters. Does ev…

I agree that data can be valuable and even one record loss can be catastrophic.

I agree that there needs to be better documentation.

I just don't agree that losing 3 replicas each living in a different DC at once is a realistic concern. The ones that would truly be concerned about this issue would do one of two things - run RF>3 (yes, it costs more) or set up some disaster recovery strategy (e.g run in multiple regions, yes that costs more.)

Because truth be told - losing 3 AZs at once is a disaster. And even if you durably persisted to disk - all 3 disks may have become corrupt anyway.

Post reply on HN