Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issues we've encountered while building a Kafka based data processing pipeline

sixfold.medium.com

Post reply on HN
Print view View original

61–70 of 96 posts

Re: Issues we've encountered while building a Kafka based data processing pipeline

#61

Post by IggleSniggle »

dwohnitmok wrote: post #57
superyesh wrote: post #53

Earlier quoted context omitted.

>My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres. Sorry thats just a clickbait-y statement. I love Postgres, try handling 100-500k rps of data coming in from various sources reading and writing to it. You are going to get bottlenecked on how many connections you can handle, you will end up throwing pgBouncers on top of it. Eventually you will run out of disk, star…

I think anotherhue would agree that half a million write requests per second counts as a valid answer to "you can't do what you need with a beefy Postgres," but that is also a minority of situations.

It's just hard to know what people mean when they say "most people don't need to do this." I was sitting wondering about a similar scale (200-1000 rps), where I've had issues with scaling rabbitmq, and have been thinking about whether kafka might help.

Without context provided, you might think: "oh, here's somebody with kafka and postgres experience, saying that postgres has some other super powers I hadn't learned about yet. Maybe I need to go learn me some more postgres and see how it's possible."

It would be helpful for folks to provide generalized measures of scale. "Right tool for the job," sure, but in the case of postgres, it often feels like there are a lot of incredible capabilities lurking.

I don't know what's normal for day-to-day software engineers anymore. Was the parent comment describing 100-500 rps really "a minority of situations?" I'm sure it is for most businesses. But is it "the minority of situations" that software engineers are actively trying to solve in 2021? I have no clue.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#62

Post by LgWoodenBadger »

anentropic wrote: post #47
LgWoodenBadger wrote: post #42

I didn't quite follow their explanation for why producing to Kafka first didn't/wouldn't work for them (db state potentially being out of sync requiring continuous messaging until fixed).

it's a chicken and egg problem you can either send a kafka message but potentially not commit the db transaction (i.e. an event is published for which the action did not actually occur) or commit the db transaction and potentially not send the kafka message it sounds like they implemented something like the Transactional Outbox pattern https://microservices.io/patterns/data/transactional-outbox.... i.e. you use the d…

Their solution seems like a "produce to Kafka first" but with extra steps.

Regarding:

When we produce first and the database update fails (because of incorrect state) it means in the worst case we enter a loop of continuously sending out duplicate messages until the issue is resolved

I don't understand where either 1) the incorrect state or 2) the need to continuously send duplicate messages come from.

Regarding:

The Job might still fail during execution, in which case it’s retried with exponential backoff, but at least no updates are lost. While the issue persists, further state change messages will be queued up also as Jobs (with same group value). Once the (transient) issue resolves, and we can again produce messages to Kafka, the updates would go out in logical order for the rest of the system and eventually everyone would be in sync.

This is the part that is equivalent to Kafka-first, except with all the extra steps of a job scheduling, grouping, tracking, and execution framework on top of it.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#63

Post by Serow225 »

IggleSniggle wrote: post #61
dwohnitmok wrote: post #57

Earlier quoted context omitted.

I think anotherhue would agree that half a million write requests per second counts as a valid answer to "you can't do what you need with a beefy Postgres," but that is also a minority of situations.

It's just hard to know what people mean when they say "most people don't need to do this." I was sitting wondering about a similar scale (200-1000 rps), where I've had issues with scaling rabbitmq, and have been thinking about whether kafka might help. Without context provided, you might think: "oh, here's somebody with kafka and postgres experience, saying that postgres has some other super powers I hadn't learned a…

that seems like an awfully low number to be running into issues with RabbitMQ ?

Re: Issues we've encountered while building a Kafka based data processing pipeline

#64

Post by yongjik »

jghn wrote: post #33
ThinkBeat wrote: post #26

Earlier quoted context omitted.

This is exactly how I feel about it. A while back I was on team building a non-critical, low volume application. It just involves people sending a message basically. (there is more too it). The consultants said he had to use Kafka because the messages could come in really fast. I said we should stick with Postgres. No, they said, we really need Kakfa to be able to handle this. Then I went and spun up Postgres on my w…

People tend to not realize how big "at scale" problems really are. Instead anything at a scale at the edge of their experience is "at scale" and they reach for the tools they've read one is supposed to use in those situations. It makes sense, people don't know what they don't know. And thus we have a world where people have business needs that could be powered by my low end laptop but solutions inspired by the megaco…

You are not thinking enterprisey enough. Everything is Big Scale if you add enough layers, because all these overheads add up, to which the solution is of course more layers.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#65

Post by nitwit005 »

fafle wrote: post #8

The issue of running a transaction that spans multiple heterogeneous systems is usually solved with a 2 phase commit. The "jobs" abstraction from the article looks similar to the "coordinator" in 2PC. The article does not talk about how they achieve fault tolerance in case the "job" crashes inbetween the two transactions. Postgres supports the XA standard, which might help with this. Kafka does not support it.

The solution I've seen is to write the message you want to send to the DB along with the transaction, and have some separate thread that tries to send the messages to Kafka.

Although, from various code bases I've seen, a lot of people just don't seem to worry about the possibility of data loss.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#66

Post by tiew9Vii »

sumtechguy wrote: post #23
BFLpL0QNek wrote: post #10

Earlier quoted context omitted.

It depends on what the events are, how they are structured. You get guaranteed ordering at the partition level. Items are partitioned by key so you also get guaranteed ordering for a key. If you have guaranteed ordering for a key you can’t get total ordering across all keys but you can get eventual consistency across the keys. Ultimately if you want ordering you have to design around being eventually consistent. I do…

For kafka the default is round robin in each partition. A hash key can let you direct the work to particular partitions. Each partition is guaranteed ordering. Also only one consumer in a consumer group can remove an item from a partition at a time. No two consumers in a consumer group will get the same message.

It's round robin if no key specified otherwise it uses murmur2 hash of the key so the partition for a key is always deterministic.

Just checking the docs it appears the round robin is no longer true after Confluent Platform 5.4. After 5.4 it looks like if no key specified the partition is assigned based on the batch being processed.

> If the key is provided, the partitioner will hash the key with murmur2 algorithm and divide it by the number of partitions. The result is that the same key is always assigned to the same partition. If a key is not provided, behavior is Confluent Platform version-dependent:...

https://docs.confluent.io/platform/current/clients/producer....

Re: Issues we've encountered while building a Kafka based data processing pipeline

#68

Post by staticassertion »

(1) seems best solved by having a 'on heavy task, publish to a secondary topic'. This is good if you have flaky messages that need to be retried in the background, without blocking all of your 'good' messages.

(2) this problem should be avoided in general by just having idempotent services. Just as a hard restriction, forever, build services to be idempotent. It should be the exception to have a non-idempotent service, and it should be carefully understood.

That said, if you have (1) as a consistent issue, like if every message is flaky, kafka isn't the right solution. Postgres-based queues are perfect for this because you can examine the table as a whole, making more informed decisions about what you want to process (or not process).

Re: Issues we've encountered while building a Kafka based data processing pipeline

#70

Post by anotherhue »

superyesh wrote: post #53
anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

>My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres. Sorry thats just a clickbait-y statement. I love Postgres, try handling 100-500k rps of data coming in from various sources reading and writing to it. You are going to get bottlenecked on how many connections you can handle, you will end up throwing pgBouncers on top of it. Eventually you will run out of disk, star…

Indeed! That sounds absolutely like it requires a real pub/sub. 'Actually needing it' is the exception in my experience though.
Post reply on HN