Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

101–110 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#101

Post by Groxx »

twodave wrote: post #66
Groxx wrote: post #24

Earlier quoted context omitted.

having never hosted a GraphQL service, but I can see many obvious room for problems: is there some reason GraphQL gets so much hate? it always feels to me like it's mostly just a normal RPC system but with some incredibly useful features (pipelining, and super easy to not request data you don't need), with obvious perf issues in code and obvious room for perf abuse because it's easy to allow callers to do N+1 nonsens…

As someone who works with GraphQL daily, many of the criticisms out there are from before the times of persisted queries, query cost limits, and composite schemas. It’s a very mature and useful technology. I agree with it maybe being less suitable for a public API, but less because of possible abuse and more because simple HTTP is a lot more widely known. It depends on the context, as in all things, of course.

yeah, I took one look at it and said "great, so add some cost tracking and kill requests before they exceed it" because like. obviously. it's similar to exposing a SQL endpoint: you need to build for that up front or the obvious results will happen.

which I fully understand is more work than "it's super easy just X" which it gets presented as, but that's always the cost of super flexible things. does graphql (or the ecosystem, as that's part of daily life of using it) make that substantially worse somehow? because I've dealt with people using protobuf to avoid graphql, then trying to reimplement parts of its features, and the resulting API is always an utter abomination.

Re: Kafka is Fast – I'll use Postgres

#102

Post by j45 »

sneilan1 wrote: post #9

I'm starting to like mongodb a lot more given the python library mongomock. I find it wonderful to create tests that run my queries against mongo in code before I deploy them. Yes, mongo has a lot of quirks and you have to know aws networking to set it up with your vpc so you don't get nailed with egress costs. And it's not the same query patterns and some queries are harder and you have maintain your own schemas. Bu…

That might work for some.

I prefer not to start with a nosql database and then undertake odysseys to make it into a relational database.

Re: Kafka is Fast – I'll use Postgres

#103

Post by PeterCorless »

kragen wrote: post #80
PeterCorless wrote: post #43

Earlier quoted context omitted.

Exactly. Just yesterday someone posted how they can do 250k messages/second with Redpanda (Kafka-compatible implementation) on their laptop . https://www.youtube.com/watch?v=7CdM1WcuoLc Getting even less than that throughput on 3x c7i.24xlarge — a total of 288 vCPUs – is bafflingly wasteful. Just because you can do something with Postgres doesn't mean you should. > 1. One camp chases buzzwords. > 2. The other camp ch…

This sounded interesting to me, and it looks like the plan is to make Redpanda open-source at some point in the future, but there's no timeline: https://github.com/redpanda-data/redpanda/tree/dev/licenses

Correct. Redpanda is source-available.

When you have C++ code, the number of external folks who want to — and who can effectively, actively contribute to the code — drops considerably. Our "cousins in code," ScyllaDB last year announced they were moving to source-available because of the lack of OSS contributors:

> Moreover, we have been the single significant contributor of the source code. Our ecosystem tools have received a healthy amount of contributions, but not the core database. That makes sense. The ScyllaDB internal implementation is a C++, shard-per-core, future-promise code base that is extremely hard to understand and requires full-time devotion. Thus source-wise, in terms of the code, we operated as a full open-source-first project. However, in reality, we benefitted from this no more than as a source-available project.

Source: https://www.scylladb.com/2024/12/18/why-were-moving-to-a-sou...

People still want to get free utility of the source-available code. Less commonly they want be able to see the code to understand it and potentially troubleshoot it. Yet asking for active contribution is, for almost all, a bridge too far.

Re: Kafka is Fast – I'll use Postgres

#104

Post by adamtulinius »

uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

I remember doing 900k writes/s (non-replicated) already back on kafka 0.8 with a random physical server with an old fusionio drive (says something about how long ago this was :D).

It's a fair point that if you already have a pgsql setup, and only need a few messages here and there, then pg is fine. But yeah, the 96 vcpu setup is absurd.

Re: Kafka is Fast – I'll use Postgres

#105

Post by losvedir »

Maybe I missed it in the design here, but this pseudo-Kafka Postgres implementation doesn't really handle consumer groups very well. The great thing about Kafka consumer groups is it makes it easy to spread the load over several instances running your service. They'll all connect using the same group, and different partitions will be assigned to the different instances. As you scale up or down, the partition responsibilities will be updated accordingly.

You need some sort of server-side logic to manage that, and the consumer heartbeats, and generation tracking, to make sure that only the "correct" instances can actually commit the new offsets. Distributed systems are hard, and Kafka goes through a lot of trouble to ensure that you don't fail to process a message.

Re: Kafka is Fast – I'll use Postgres

#106

Post by adamtulinius »

blenderob wrote: post #96
uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

> Has this person actually benchmarked kafka? Is anyone actually reading the full article, or just reacting to the first unimpressive numbers you can find and then jumping on the first dismissive comment you can find here? Benchmarking Kafka isn't the point here. The author isn't claiming that Postgres outperforms Kafka. The argument is that Postgres can handle modest messaging workloads well enough for teams that do…

The problem is benchmarking on the 96 vcpu server, because at that point the author seems to miss the point of Kafka. That's just a waste of money for that performance.

Re: Kafka is Fast – I'll use Postgres

#107

Post by j45 »

kitd wrote: post #71
j45 wrote: post #51

Earlier quoted context omitted.

Is it about what Kafka could get or what you need right now. Kafka is a full on steaming solution. Postgres isn’t a buzzword. It can be a capable placeholder until it’s outgrown. One can arrive at Kafka with a more informed run history from Postgres.

> Kafka is a full on steaming solution. Freudian slip? ;)

Haha, and a typo!

Re: Kafka is Fast – I'll use Postgres

#108

Post by kragen »

PeterCorless wrote: post #103
kragen wrote: post #80

Earlier quoted context omitted.

This sounded interesting to me, and it looks like the plan is to make Redpanda open-source at some point in the future, but there's no timeline: https://github.com/redpanda-data/redpanda/tree/dev/licenses

Correct. Redpanda is source-available. When you have C++ code, the number of external folks who want to — and who can effectively, actively contribute to the code — drops considerably. Our "cousins in code," ScyllaDB last year announced they were moving to source-available because of the lack of OSS contributors: > Moreover, we have been the single significant contributor of the source code. Our ecosystem tools have…

Right, open source is generally of benefit to users, not to the author, and users do get some of that benefit from being able to see the source. I wouldn't want to look at it myself, though, for legal reasons.

Re: Kafka is Fast – I'll use Postgres

#109

Post by mrkeen »

loftsy wrote: post #46

I am about to start a project. I know I want an event sourced architecture. That is, the system is designed around a queue, all actors push/pull into the queue. This article gives me some pause. Performance isn't a big deal for me. I had assumed that Kafka would give me things like decoupling, retry, dead-lettering, logging, schema validation, schema versioning, exactly once processing. I like Postgres, and obviously…

Event-sourcing != queue.

Event-sourcing is when you buy something and get a receipt, you go stick it in a shoe-box for tax time.

A queue is you get given receipts, and you look at them in the correct order before throwing each one away.

Re: Kafka is Fast – I'll use Postgres

#110

Post by jasonthorsness »

Using a single DBMS for many purposes because it is so flexible and “already there” from an operations perspective is something I’ve seen over and over again. It usually goes wrong eventually with one workload/use screwing up others but maybe that’s fine and a normal part of scaling?

I think a bigger issue is the DBMS themselves getting feature after feature and becoming bloated and unfocused. Add the thing to Postgres because it is convenient! At least Postgres has a decent plugin approach. But I think more use cases might be served by standalone products than by add-ons.

Post reply on HN