Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issues we've encountered while building a Kafka based data processing pipeline

sixfold.medium.com

Post reply on HN
Print view View original

41–50 of 96 posts

Re: Issues we've encountered while building a Kafka based data processing pipeline

#41

Post by LoriP »

dionian wrote: post #31
anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

Honest question, how do you expire content in Postgres? Every time I start to use it for ephemeral data I start to wonder if I should have used TimescaleDB or if I should be using something else...

You likely already know that TimescaleDB is an extension to PostgreSQL, so you get everything you'd get with PostgreSQL plus the added goodies of TimescaleDB. All that said, you can drop (or detach) data partitions in PostgreSQL (however you decide to partition...) Does that not do the trick for your use case, though? https://www.postgresql.org/docs/10/ddl-partitioning.html

Transparency: I work for Timescale

Re: Issues we've encountered while building a Kafka based data processing pipeline

#43

Post by mrweasel »

anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

We work with a client who has requested a Kafka cluster. They can’t really say why, or what they need it for, but the now have a very large cluster which doesn’t do much. I know why they want it, same reason why they “need” Kubernetes. So far they use it as a sort of message bus.

It’s not that there’s anything wrong with Kafka, it’s a very good product and extremely robust. Same with Kubernetes, it has it uses and I can’t fault anyone for having it as a consideration.

My problem is when people ignore how capable modern servers are, and when developers don’t see the risks in building these highly complex systems, if something much simpler would solve the same problem, only cheaper and safer.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#44

Post by kitd »

HelloNurse wrote: post #24

The only time I used Kafka, it was involuntarily (included for the sake of fashion in some complicated IBM product, where it hid among WebSphere, DB2 and other bigger elephants) and it ran my server out of disk space because due to a bug ridiculously massive temporary files weren't erased. Needless to say, I wasn't impressed: just one more hazard to worry about.

due to a bug

Data retention time is Kafka config 101. Are you sure it was a bug?

Re: Issues we've encountered while building a Kafka based data processing pipeline

#45

Post by jerf »

jghn wrote: post #33
ThinkBeat wrote: post #26

Earlier quoted context omitted.

This is exactly how I feel about it. A while back I was on team building a non-critical, low volume application. It just involves people sending a message basically. (there is more too it). The consultants said he had to use Kafka because the messages could come in really fast. I said we should stick with Postgres. No, they said, we really need Kakfa to be able to handle this. Then I went and spun up Postgres on my w…

People tend to not realize how big "at scale" problems really are. Instead anything at a scale at the edge of their experience is "at scale" and they reach for the tools they've read one is supposed to use in those situations. It makes sense, people don't know what they don't know. And thus we have a world where people have business needs that could be powered by my low end laptop but solutions inspired by the megaco…

The GHz aspect of Moore's law died over a decade ago, and I suppose it's fair to say most other stuff has also slowed down, but if you've got a job that is embarrassingly parallel, which a lot of these "big data" jobs are, people badly underestimate how much progress there has been in the server space even so in the last 10 years if they're not paying attention. What was "big data" in 2011 can easily be "spin up a single 32-core instance with 1TB RAM for a couple of hours" in 2021. Even beyond the "big data" that my laptop comfortably handles.

I'm slowly wandering into a data science role lately, and I've been dealing with teams who are all kinds of concerned about whether or not we can handle the sheer, overwhelming volume of their (summary) data. "Well, let's see, how much data are we talking about?" "Oh, gosh, we could generate 200 or 300 megabytes a day." (Of uncompressed JSON.) Well, you know, if I have to bust out a second Raspberry Pi I'll be sure to charge it to your team.

The funny thing is that some of these teams have the experience that they ought to know better. They are legitimately running cloud services with dozens of large nodes continually running at high utilization and chewing through gigabytes of whatever per second. In their own worlds they would absolutely know that a couple hundred megabytes is nothing. They'll often have known places in their stack where they burn through a few hundred megabytes in internal API calls or something unnecessarily, and it will barely rise to the level of a P3 bug, quite legitimately so. But when they start thinking in terms of (someone else's) databases it's like they haven't updated their sense of size since 2005.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#46

Post by lmilcin »

anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

Just because you can do it with Postgres doesn't mean it is the best tool for the job.

Sometimes the restrictions placed on the user are as important. Kafka presents a specific interface to the user that causes users to build their applications in certain way.

While you can replicate almost all functionality of Kafka with Postgres (except for performance, but hardly anybody needs as much of it), we all know what we end up with when we set up Postgres and use it to integrate applications with each other.

If developers had discipline they could of course crate tables with appendable logs of data, marked with a partition, that consumers could process from with basically same guarantees as with Kafka.

But that is not how it works in reality.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#47

Post by anentropic »

LgWoodenBadger wrote: post #42

I didn't quite follow their explanation for why producing to Kafka first didn't/wouldn't work for them (db state potentially being out of sync requiring continuous messaging until fixed).

it's a chicken and egg problem

you can either send a kafka message but potentially not commit the db transaction (i.e. an event is published for which the action did not actually occur) or commit the db transaction and potentially not send the kafka message

it sounds like they implemented something like the Transactional Outbox pattern https://microservices.io/patterns/data/transactional-outbox....

i.e. you use the db transaction to also commit a record of your intent to send a kafka message - you can then move the actual event sending to a separate process and implement at-least-once semantics

This is the job queueing system they described in the article

Re: Issues we've encountered while building a Kafka based data processing pipeline

#48

Post by mfateev »

temporal.io provides much higher level abstraction for building asynchronous microservices. It allows one to model async invocations as synchronous blocking calls of any duration (months for example). And the state updates and queueing are transactional out of the box.

Here is an example using Typescript SDK:

   async function main(userId, intervals){ 
     // Send reminder emails, e.g. after 1, 7, and 30 days
   
     for (const interval of intervals) {
       await sleep(interval * DAYS);
       // can take hours if the downstream service is down
       await activities.sendEmail(interval, userId); 
     } 
     // Easily cancelled when user unsubscribes
   }
Disclaimer: I'm one of the creators of the project.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#49

Post by geodel »

kitd wrote: post #44
HelloNurse wrote: post #24

The only time I used Kafka, it was involuntarily (included for the sake of fashion in some complicated IBM product, where it hid among WebSphere, DB2 and other bigger elephants) and it ran my server out of disk space because due to a bug ridiculously massive temporary files weren't erased. Needless to say, I wasn't impressed: just one more hazard to worry about.

due to a bug Data retention time is Kafka config 101. Are you sure it was a bug?

Considering how half-assed Kafka is in general, that it needs all clients code changes when Kafka servers are upgraded. It is very likely that user hit Kafka bug.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#50

Post by emerongi »

BiteCode_dev wrote: post #40
sumtechguy wrote: post #21

Earlier quoted context omitted.

I have done both patterns. Kafka has its use case. Databases have theirs. You can make a DB do what kafka does. But you also add in the programming overhead of getting the DB semantics correct to make an event system. When I see people saying 'lets put the DB into kafka' I make the exact same argument. You will spend more time making kafka act like a database and getting the semantics right. Kafka is more of a data/e…

Well, if you want events, you have lighter alternative. Redis pub/sub, crossbar... Even rabbitMQ is lighter. If you need queuing, you have libs like celery. You don't need to go full kafka

Redis - it's probably in your stack already and fills all the use-cases.
Post reply on HN