Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

281–290 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#281

Post by EdwardDiego »

AtlasBarfed wrote: post #233
agentultra wrote: post #6

You have to be careful with the approach of using Postgres for everything. The way it locks tables and rows and the serialization levels it guarantees are not immediately obvious to a lot of folks and can become a serious bottle-neck for performance-sensitive workloads. I've been a happy Postgres user for several decades. Postgres can do a lot! But like anything, don't rely on maxims to do your engineering for you.

Postgres is just fantastic software. But anytime you treat a database, or a queue, like a black box dumpster, problems will ensue.

Exactly. Or worse, you treat one as a straightforward black box swap in replacement for another. If you're looking to scale, you _will_ need to code to the idiosyncraties of your chosen solution.

Re: Kafka is Fast – I'll use Postgres

#282

Post by EdwardDiego »

kragen wrote: post #100
PeterCorless wrote: post #69

Earlier quoted context omitted.

But in this case, it is like saying "You don't need a fuel truck. You can transport 9,000 gallons of gasoline between cities by gathering 9,000 1-gallon milk jugs and filling each, then getting 4,500 volunteers to each carry 2 gallons and walk the entire distance on foot." In this case, you do just need a single fuel truck. That's what it was built for. Avoiding using a design-for-purpose tool to achieve the same res…

Getting a 288-core machine might be easier than setting up Kafka; I'm guessing that it would be a couple of weeks of work to learn enough to install Kafka the first time. Installing Postgres is trivial.

Just use Strimzi if you're in a K8s world (disclosure used to work on Strimzi for RH, but I still think it's far better than Helm charts or fully self-managed, and far cheaper than fully managed).

Re: Kafka is Fast – I'll use Postgres

#283

Post by EdwardDiego »

freedomben wrote: post #149
brianmcc wrote: post #115

Earlier quoted context omitted.

"Lots of the team knows Postgres really well, nobody knows Kafka at all yet" is also an underrated factor in making choices. "Kafka was the ideal technical choice but we screwed up the implementation through well-intentioned inexperience" being an all too plausible outcome.

Indeed, I've seen this happen first hand where there was really only one guy who really "knew" Kafka, and it was too big of a job for just him. In that case it was fine until he left the company, and then it became a massive albatross and a major pain point. In another case, the eng team didn't really have anyone who really "knew" Kafka but used a managed service thinking it would be fine. It was until it wasn't, and…

I'm wondering why there wasn't any push for the Kafka guy to share his knowledge within his team, or to other teams?

Re: Kafka is Fast – I'll use Postgres

#284

Post by dagss »

shikhar wrote: post #57

Postgres is a way better fit than Kafka if you want a large number of durable streams. But a flexible OLTP database like PG is bound to require more resources and polling loops (not even long poll!) are not a great answer for following live updates. Plug: If you need granular, durable streams in a serverless context, check out s2.dev

s2.dev looks cool... I jumped around the home page a bit and couldn't perfectly grasp what it is quickly though. But if it is about decoupling the Kafka approach and client side libraries from the use of Kafka specifically I am cheering for you.

Could you see using the s2.dev protocol on top of services using SQL in the way of the article, assigning event sequence numbers, as a good fit? Or is s2 fundamentally the component that assigns event numbers?

I feel like we tried to do something similar to you, but for SQL DBs, but am not sure:

https://github.com/vippsas/feedapi-spec

Re: Kafka is Fast – I'll use Postgres

#285

Post by EdwardDiego »

PeterCorless wrote: post #139
kragen wrote: post #130

Earlier quoted context omitted.

I mean, setting up Zookeeper, tweaking the kernel settings, configuring the hardware, the kind of stuff mentioned in guides like https://medium.com/@ankurrana/things-nobody-will-tell-you-se... and https://dungeonengineering.com/the-kafkaesque-nightmare-of-m... . Apparently you can do without Zookeeper now, but that's another choice to make, possibly doing careful experiments with both choices to see what's better. Mu…

True. Redpanda does not use Zookeeper. Yet to also be fair to the Kafka folks, Zookeeper is no longer default and hasn't been since April 2025 with the release of Apache Kafka 4.0: "Kafka 4.0's completed transition to KRaft eliminates ZooKeeper (KIP-500), making clusters easier to operate at any scale." Source: https://developer.confluent.io/newsletter/introducing-apache...

Good on you for being fair in this discussion :)

Re: Kafka is Fast – I'll use Postgres

#286

Post by EdwardDiego »

codeflo wrote: post #280
EdwardDiego wrote: post #271

Earlier quoted context omitted.

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

Do you mean this in the sense that listeners don't remove messages, as one would expect from a queue data structure?

Exactly. There's no concept in Kafka (yet...) of "acking" or DLQs, Kafka is very good at what it does by being deliberately stupid, it knows nothing about your messages or who has consumed them and who hasn't.

That was all deliberately pushed onto consumers to manage to achieve scale.

Re: Kafka is Fast – I'll use Postgres

#287

Post by jen20 »

codeflo wrote: post #280
EdwardDiego wrote: post #271

Earlier quoted context omitted.

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

Do you mean this in the sense that listeners don't remove messages, as one would expect from a queue data structure?

That is the major difference - clients track their read offsets rather than the structure removing messages. There aren't really "listeners" in the sense of a pub-sub?

Re: Kafka is Fast – I'll use Postgres

#288

Post by dagss »

j45 wrote: post #92
dzonga wrote: post #81

what's not spoken about in the above article ? ease of use. in ruby If I want to use kafka I can use karafka. or redis streams via the redis library. likewise if kafka is too complex to run there's countless alternatives which work as well - hell even 0mq with client libraries. now with the postgres version I have to write my own stuff which I might not where it's gonna lead me. postgres is scalable, no one doubts th…

I’m not sure where it says you have to write your own stuff, there seem to be some of queues with libraries. https://github.com/dhamaniasad/awesome-postgres There is at least a Python example here.

Work queues is easy.

It is significantly more work for the client side implementation of event log consumers, which the article also talk about. For instance persisting the client side cursors. And I have not seen widely used standard implementations of those. (I started one myself once but didn't finish.)

Re: Kafka is Fast – I'll use Postgres

#289

Post by nasretdinov »

codeflo wrote: post #280
EdwardDiego wrote: post #271

Earlier quoted context omitted.

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

Do you mean this in the sense that listeners don't remove messages, as one would expect from a queue data structure?

Well, it's impractical to try to handle messages individually in Kafka, it's designed to acknowledge entire batches (since it's a distributed append-only log). You can still do that, but the performance will be no better than an SQL database

Re: Kafka is Fast – I'll use Postgres

#290

Post by dagss »

javier2 wrote: post #209
munchbunny wrote: post #119

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…

I dont disagree, and I am trying to argue for it myself, and have used postgres as a "queue" or the backlog of events to be sent (like outbox pattern). But what if I have 4 services that needs to know X happened to customer Y? I feel like it quickly becomes cumbersome with a postgres event delivery to make sure everyone gets the events they need delivered. The posted link tries to address this at least.

The standard approach, which Kafka also uses beneath all the libraries hiding it from you, is:

The publisher has a set of tables (topics and partitions) of events, ordered and with each event having an assigned event sequence number.

Publisher stores no state for consumers in any way.

Instead, each consumer keeps a cursor (a variable holding an event sequence number) indicating how far it has read for each event log table it is reading.

Consumer can then advance (or rewind) its own cursor in whatever way it wishes. The publisher is oblivious to any consumer side state.

This is the fundamental piece of how event log publishing works (as opposed to queues which is something else entirely; and the article talks about both usecases).

Post reply on HN