Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

71–80 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#71

Post by kitd »

j45 wrote: post #51
PeterCorless wrote: post #43

Earlier quoted context omitted.

Exactly. Just yesterday someone posted how they can do 250k messages/second with Redpanda (Kafka-compatible implementation) on their laptop . https://www.youtube.com/watch?v=7CdM1WcuoLc Getting even less than that throughput on 3x c7i.24xlarge — a total of 288 vCPUs – is bafflingly wasteful. Just because you can do something with Postgres doesn't mean you should. > 1. One camp chases buzzwords. > 2. The other camp ch…

Is it about what Kafka could get or what you need right now. Kafka is a full on steaming solution. Postgres isn’t a buzzword. It can be a capable placeholder until it’s outgrown. One can arrive at Kafka with a more informed run history from Postgres.

> Kafka is a full on steaming solution.

Freudian slip? ;)

Re: Kafka is Fast – I'll use Postgres

#72

Post by j45 »

loftsy wrote: post #46

I am about to start a project. I know I want an event sourced architecture. That is, the system is designed around a queue, all actors push/pull into the queue. This article gives me some pause. Performance isn't a big deal for me. I had assumed that Kafka would give me things like decoupling, retry, dead-lettering, logging, schema validation, schema versioning, exactly once processing. I like Postgres, and obviously…

It might look like a lot of effort, but if you follow a tutorial/YouTube video step by step you will be surprised.

It’s mostly registering the Postgres database functions which is one time.

There are also pre-made Postgres extensions that already run the queue.

These days i would like consider m starting with Supabase self hosted which has the Postgres ready to tweak.

Re: Kafka is Fast – I'll use Postgres

#73

Post by SoftTalker »

agentultra wrote: post #6

You have to be careful with the approach of using Postgres for everything. The way it locks tables and rows and the serialization levels it guarantees are not immediately obvious to a lot of folks and can become a serious bottle-neck for performance-sensitive workloads. I've been a happy Postgres user for several decades. Postgres can do a lot! But like anything, don't rely on maxims to do your engineering for you.

This is true of any data storage. You have to understand the concurrency model and assumptions, and know where bottlenecks can happen. Even among relational databases there are significant differences.

Re: Kafka is Fast – I'll use Postgres

#74

Post by justinhj »

As engineers we should try to use the right tool for the job, which means thinking about the development team's strengths and weaknesses as well as differentiating factors your product should focus on. Often we are working in the cloud and it's much easier to use a queue or a log database service than manage a bunch of sql servers and custom logic. It can be more cost effective too once you factor in the development time and operational costs.

The fact that there is no common library that implements the authors strategy is a good sign that there is not much demand for this.

Re: Kafka is Fast – I'll use Postgres

#75

Post by j45 »

this_user wrote: post #59

The real two camps seem to be: 1) People constantly chasing the latest technology with no regard for whether it's appropriate for the situation. 2) People constantly trying to shoehorn their favourite technology into everything with no regard for whether it's appropriate for the situation.

Kafka is anything but new. It does get shoehorned too.

Postgres also has been around for a long time and a lot of people didn’t know all it can do which isn’t what we normally think about with a database.

Appropriateness is a nice way to look at it as long as it’s clear whether or not it’s about personal preferences and interpretations and being righteous towards others with them.

Customers rarely care about the backend or what it’s developed in, except maybe for developer products. It’s a great way to waste time though.

Re: Kafka is Fast – I'll use Postgres

#76

Post by singron »

vbezhenar wrote: post #27

How do you implement "unique monotonically-increasing offset number"? Naive approach with sequence (or serial type which uses sequence automatically) does not work. Transaction "one" gets number "123", transaction "two" gets number "124". Transaction "two" commits, now table contains "122", "124" rows and readers can start to process it. Then transaction "one" commits with its "123" number, but readers already past "…

The log_counter table tracks this. It's true that a naive solution using sequences does not work for exactly the reason you say.

Re: Kafka is Fast – I'll use Postgres

#77

Post by mxey »

PeterCorless wrote: post #43
uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

Exactly. Just yesterday someone posted how they can do 250k messages/second with Redpanda (Kafka-compatible implementation) on their laptop . https://www.youtube.com/watch?v=7CdM1WcuoLc Getting even less than that throughput on 3x c7i.24xlarge — a total of 288 vCPUs – is bafflingly wasteful. Just because you can do something with Postgres doesn't mean you should. > 1. One camp chases buzzwords. > 2. The other camp ch…

Doesn’t Kafka/Redpanda have to fsync for every message?

Re: Kafka is Fast – I'll use Postgres

#78

Post by School-Cotton »

greenavocado wrote: post #42
dangoodmanUT wrote: post #36

96 cores to get 240MB/s is terrible. Redpanda can do this with like one or two cores

Redpanda might be good (I don't know) but I threw up a little in my mouth when I opened their website and saw "Build the Agentic Data Plane"

The marketing website of every data-related startup sounds like that now. I agree it’s dumb, but you can safely ignore it.

Re: Kafka is Fast – I'll use Postgres

#79

Post by PeterCorless »

this_user wrote: post #59

The real two camps seem to be: 1) People constantly chasing the latest technology with no regard for whether it's appropriate for the situation. 2) People constantly trying to shoehorn their favourite technology into everything with no regard for whether it's appropriate for the situation.

2) above is basically "Give a kid a hammer, and everything becomes a nail."

The third camp:

3) People who look at a task, then apply a tool appropriate for the task.

Re: Kafka is Fast – I'll use Postgres

#80

Post by kragen »

PeterCorless wrote: post #43
uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

Exactly. Just yesterday someone posted how they can do 250k messages/second with Redpanda (Kafka-compatible implementation) on their laptop . https://www.youtube.com/watch?v=7CdM1WcuoLc Getting even less than that throughput on 3x c7i.24xlarge — a total of 288 vCPUs – is bafflingly wasteful. Just because you can do something with Postgres doesn't mean you should. > 1. One camp chases buzzwords. > 2. The other camp ch…

This sounded interesting to me, and it looks like the plan is to make Redpanda open-source at some point in the future, but there's no timeline: https://github.com/redpanda-data/redpanda/tree/dev/licenses
Post reply on HN