Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

161–170 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#161

Post by lenkite »

blenderob wrote: post #114
uberduper wrote: post #68

Earlier quoted context omitted.

It seems like their point was to criticize people for using new tech instead of hacking together unscalable solutions with their preferred database.

That wasn't their point. Instead of posting snarky comments, please review the site guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize."

But honestly, isn't that the strongest plausible interpretation according to the "site guidelines" ? When one explicitly says that the one camp chases "buzzwords" and the other chases "common sense", how else are you supposed to interpret it ?

Re: Kafka is Fast – I'll use Postgres

#162

Post by apetrov »

roozbeh18 wrote: post #67
uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

Just checked my single node Kafka setup which currently handles 695.27k e/s (average daily) into elasticsearch without breaking a sweat. kafka has been the only stable thing in this whole setup. zeek -> kafka -> logstash -> elastic

out of curiosity, what does your service do that it handles almost 700K events/sec?

Re: Kafka is Fast – I'll use Postgres

#163

Post by sneilan1 »

zenmac wrote: post #148
sneilan1 wrote: post #13

Earlier quoted context omitted.

Yes, performance can be a big issue with postgres. And vertical scaling can really put a damper on things when you have a major traffic hit. Using it for kafka is misunderstanding the one of the great uses of kafka which is to help deal with traffic bursts. All of a sudden your postgres server is overwhelmed and the kafka server would be fine.

>And vertical scaling can really put a damper on things when you have a major traffic hit. Wouldn't OrioleDB solve that issue though?

Not familiar with OrioleDB. I’ll look it up. May I ask how this helps? Just curious.

Re: Kafka is Fast – I'll use Postgres

#164

Post by rplnt »

PeterCorless wrote: post #103
kragen wrote: post #80

Earlier quoted context omitted.

This sounded interesting to me, and it looks like the plan is to make Redpanda open-source at some point in the future, but there's no timeline: https://github.com/redpanda-data/redpanda/tree/dev/licenses

Correct. Redpanda is source-available. When you have C++ code, the number of external folks who want to — and who can effectively, actively contribute to the code — drops considerably. Our "cousins in code," ScyllaDB last year announced they were moving to source-available because of the lack of OSS contributors: > Moreover, we have been the single significant contributor of the source code. Our ecosystem tools have…

You can be open source and not take contributions. This argument doesn't make sense to me. Just stop doing the expensive part and keep the license as is.

Re: Kafka is Fast – I'll use Postgres

#165

Post by ryandvm »

I think my only complaint about Kafka is the widespread misunderstanding that it is a suitable replacement for a work queue. I should not be having to explain to an enterprise architect the distinction between a distributed work queue and event streaming platform.

Re: Kafka is Fast – I'll use Postgres

#166

Post by rileymichael »

loftsy wrote: post #46

I am about to start a project. I know I want an event sourced architecture. That is, the system is designed around a queue, all actors push/pull into the queue. This article gives me some pause. Performance isn't a big deal for me. I had assumed that Kafka would give me things like decoupling, retry, dead-lettering, logging, schema validation, schema versioning, exactly once processing. I like Postgres, and obviously…

if you need a durable log (which it sounds like you do for if you're going with event sourcing) that has those features, i'd suggest apache pulsar. you effectively get streams with message queue semantics (per-message acks, retries, dlq, etc.) from one system. it supports many different 'subscription types', so you can use it for a bunch of different use cases. running it on your own is a bit of a beast though and there's really only one hosted provider in the game (streamnative)

note that kafka has recently started investing into 'queues' in KIP-932, but they're still a long way off from implementing all of those features.

Re: Kafka is Fast – I'll use Postgres

#168

Post by joaohaas »

Sparkyte wrote: post #167

You can also use Redis as a queue if the data isn't in danger of being too important.

Even if the data is important, you can enable WAL and make sure the worker/consumer gets items by RPOPLPUSHing to a working queue. This way you can easily requeue the data if the worker ever goes offline mid-process.

Re: Kafka is Fast – I'll use Postgres

#169

Post by mxey »

zozbot234 wrote: post #151
mrkeen wrote: post #131

Earlier quoted context omitted.

The tradeoff here is that Kafka will still work perfectly if one of its instances goes down. (Or you take it down, for upgrades, etc.) Can you lose one Postgres instance?

AIUI Postgres has high-availability out of the box, so it's not a big deal to "lose" one as long as a secondary can take over.

Only replication is built-in, you need to add a cluster manager like Patroni to make it highly-available.

Re: Kafka is Fast – I'll use Postgres

#170

Post by mxey »

UltraSane wrote: post #132
mxey wrote: post #77

Earlier quoted context omitted.

Doesn’t Kafka/Redpanda have to fsync for every message?

On enterprise grade storage writes go to NVRAM buffers before being flushed to persistent storage so this isn't much of a bottleneck.

The context was somebody doing this on their laptop.
Post reply on HN