Issues we've encountered while building a Kafka based data processing pipeline
1–10 of 96 posts
Re: Issues we've encountered while building a Kafka based data processing pipeline
#2I ran a few dozen kafka clusters at MegaCorp in a previous life.
My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#3Very interested to hear how people here overcome the limits of kafka for ordered events delivery in real world, and what those were.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#4I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
using a sql db for push/pop semantic feels like using a hammer to squash a bug.. How would you model queues & partitions with ordering guarantees with pg ?
Re: Issues we've encountered while building a Kafka based data processing pipeline
#5I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
using a sql db for push/pop semantic feels like using a hammer to squash a bug.. How would you model queues & partitions with ordering guarantees with pg ?
sql has many conveniences for doing so, it wouldn't be much work.
> using a hammer to squash a bug..
Agreed - but Kafka is a much much bigger hammer. SES/Az Queues are also good choices.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#6I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
I had a great time with Kafka for prototyping. Being able to push data from a number of places, have mulitple consumers able to connect, go back and forth though time, add and remove independent consumer groups. Ran in pre-production very reliably too, for years.
But for a production-grade version of the system I'm going with SQL and, where needed, IaC-defined SQS.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#7Very interested to hear how people here overcome the limits of kafka for ordered events delivery in real world, and what those were.
I feel as if you're using Kafka and expect guaranteed ordering, then you're using the wrong tool. At best you have guaranteed ordering per partition but then you've tied your ordering/keying strategy to the amount of partitions you've enabled ... which may not ideal.
But, that's speaking from my light experience with it. I'm also curious if there's a better way :-)
Re: Issues we've encountered while building a Kafka based data processing pipeline
#8The issue of running a transaction that spans multiple heterogeneous systems is usually solved with a 2 phase commit. The "jobs" abstraction from the article looks similar to the "coordinator" in 2PC. The article does not talk about how they achieve fault tolerance in case the "job" crashes inbetween the two transactions. Postgres supports the XA standard, which might help with this. Kafka does not support it.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#9Very interested to hear how people here overcome the limits of kafka for ordered events delivery in real world, and what those were.
At lower data volumes (<10,000 events per minute) it’s perfectly feasible to just use single partition topics and then ordered event delivery is no problem at all. If a consuming service has processing times that means horizontal scaling is necessary then the topic can be repartitioned into a new topic with multiple partitions and the processing application can handle sorting the outputted data to some SLA.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#10Very interested to hear how people here overcome the limits of kafka for ordered events delivery in real world, and what those were.
It depends on what the events are, how they are structured.
You get guaranteed ordering at the partition level.
Items are partitioned by key so you also get guaranteed ordering for a key.
If you have guaranteed ordering for a key you can’t get total ordering across all keys but you can get eventual consistency across the keys.
Ultimately if you want ordering you have to design around being eventually consistent.
I don’t read a lot of papers but Leslie Lamports Time, Clocks, and the Ordering of Events in a Distributed System gave me a lot of insight in to the constraints. https://lamport.azurewebsites.net/pubs/time-clocks.pdf