Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

51–60 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#51

Post by j45 »

PeterCorless wrote: post #43
uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

Exactly. Just yesterday someone posted how they can do 250k messages/second with Redpanda (Kafka-compatible implementation) on their laptop . https://www.youtube.com/watch?v=7CdM1WcuoLc Getting even less than that throughput on 3x c7i.24xlarge — a total of 288 vCPUs – is bafflingly wasteful. Just because you can do something with Postgres doesn't mean you should. > 1. One camp chases buzzwords. > 2. The other camp ch…

Is it about what Kafka could get or what you need right now.

Kafka is a full on steaming solution.

Postgres isn’t a buzzword. It can be a capable placeholder until it’s outgrown. One can arrive at Kafka with a more informed run history from Postgres.

Re: Kafka is Fast – I'll use Postgres

#52

Post by joaohaas »

uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

Had the same thoughts, weird it didn't include Kafka numbers.

Never used Kafka myself, but we extensively use Redis queues with some scripts to ensure persistency, and we hit throughputs much higher than those in equivalent prod machines.

Same for Redis pubsubs, but those are just standard non-persistent pubsubs, so maybe that gives it an upper edge.

Re: Kafka is Fast – I'll use Postgres

#53

Post by j45 »

loire280 wrote: post #40
uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

In fact, a properly-configured Kafka cluster on minimal hardware will saturate its network link before it hits CPU or disk bottlenecks.

But it can do so many processes a second I’ll be able to scale to the moon before I ever launch.

Re: Kafka is Fast – I'll use Postgres

#54

Post by fud101 »

agentultra wrote: post #6

You have to be careful with the approach of using Postgres for everything. The way it locks tables and rows and the serialization levels it guarantees are not immediately obvious to a lot of folks and can become a serious bottle-neck for performance-sensitive workloads. I've been a happy Postgres user for several decades. Postgres can do a lot! But like anything, don't rely on maxims to do your engineering for you.

When someone says just use Postgres, are they using the same instance for their data as well for the queue?

Re: Kafka is Fast – I'll use Postgres

#55

Post by misja111 »

> One camp chases buzzwords .. the other common sense

How is it common sense to try to re-implement Kafka in Posgres? You probably need something similar but more simple. Then implement that! But if you really need something like Kafka, then .. use Kafka!

IMO the author is now making the same mistake as some Kafka evangelists that try to implement a database in Kafka.

Re: Kafka is Fast – I'll use Postgres

#56

Post by jorge-d »

fukka42 wrote: post #28
agentultra wrote: post #6

You have to be careful with the approach of using Postgres for everything. The way it locks tables and rows and the serialization levels it guarantees are not immediately obvious to a lot of folks and can become a serious bottle-neck for performance-sensitive workloads. I've been a happy Postgres user for several decades. Postgres can do a lot! But like anything, don't rely on maxims to do your engineering for you.

My strategy is to use postgres first. Get the idea off the ground and switch when postgres becomes the bottleneck. It often doesn't.

Definitely, this is also one of the direction Rails is heading[1]: provide a basis setup most of the people can use out of the box. And if needed you can always plug in more "mature" solutions afterwards.

[1] https://rubyonrails.org/2024/11/7/rails-8-no-paas-required

Re: Kafka is Fast – I'll use Postgres

#57

Post by shikhar »

Postgres is a way better fit than Kafka if you want a large number of durable streams. But a flexible OLTP database like PG is bound to require more resources and polling loops (not even long poll!) are not a great answer for following live updates.

Plug: If you need granular, durable streams in a serverless context, check out s2.dev

Re: Kafka is Fast – I'll use Postgres

#58

Post by theK »

loire280 wrote: post #40
uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

In fact, a properly-configured Kafka cluster on minimal hardware will saturate its network link before it hits CPU or disk bottlenecks.

Isn't that true for everything on the cloud? I thought we are long into the era where your disk comes over the network there.

Re: Kafka is Fast – I'll use Postgres

#59

Post by this_user »

The real two camps seem to be:

1) People constantly chasing the latest technology with no regard for whether it's appropriate for the situation.

2) People constantly trying to shoehorn their favourite technology into everything with no regard for whether it's appropriate for the situation.

Re: Kafka is Fast – I'll use Postgres

#60

Post by binarymax »

ownagefool wrote: post #29

The camps are wrong. There's poles. 1. Is folks constantly adopting the new tech, whatever the motivation, and 2. I learned a thing and shall never learn anything else, ever. Of course nobody exists actually on either pole, but the closer you are to either, the less pragmatic you are likely to be.

This is it right here. My foil is the Elasticsearch replacement because PG has inverted indices. The ergonomics and tunability of these in PG are terrible compared to ES. Yes, it will search, but I wouldn’t want to be involved in constructing or maintaining that search.
Post reply on HN