Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

271–280 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#271

Post by EdwardDiego »

munchbunny wrote: post #119

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

Re: Kafka is Fast – I'll use Postgres

#272

Post by dagss »

munchbunny wrote: post #187
vbezhenar wrote: post #27

How do you implement "unique monotonically-increasing offset number"? Naive approach with sequence (or serial type which uses sequence automatically) does not work. Transaction "one" gets number "123", transaction "two" gets number "124". Transaction "two" commits, now table contains "122", "124" rows and readers can start to process it. Then transaction "one" commits with its "123" number, but readers already past "…

I have this problem in the system I work on - the short nuance-less answer from my experience is that, once your scale gets large enough, you can't prevent ordering issues entirely and you have to build the resilience into the architecture and the framing of the problem. You often end up paying for consistency with latency.

I think you may be talking past each other. In the approach taken in the article and the parent comment, if the event sequence number allocation of the writer races the reader cursor position in the wrong way, events will NEVER BE DELIVERED.

So it is a much more serious issue at stake here than event ordering/consistency.

As it happens, if you use event log tables in SQL "the Kafka way" you actually get guarantee on event ordering too as a side effect, but that is not the primary goal.

More detailed description of problem:

https://github.com/vippsas/mssql-changefeed/blob/main/MOTIVA...

Re: Kafka is Fast – I'll use Postgres

#273

Post by sarchertech »

DelaneyM wrote: post #213
oulipo2 wrote: post #124

Earlier quoted context omitted.

I want to rewrite some of my setup, we're doing IoT, and I was planning on MQTT -> Redpanda (for message logs and replay, etc) -> Postgres/Timescaledb (for data) + S3 (for archive) (and possibly Flink/RisingWave/Arroyo somewhere in order to do some alerting/incrementally updated materialized views/ etc) this seems "simple enough" (but I don't have any experience with Redpanda) but is indeed one more moving part compa…

My suggestion would be even simpler: MQTT -> Postgres (+ S3 for archive) > 1. my "fear" would be that if I use the same Postgres for the queue and for my business database... This is a feature, not a bug. In this way you can pair the handling of the message with the business data changes which result in the same transaction. This isn't quite "exactly-once" handling, but it's really really close! > 2. also that since…

> This is a feature, not a bug. In this way you can pair the handling of the message with the business data changes which result in the same transaction.

That’s a particularly nasty trap. Devs will start using this everywhere and it makes it very hard to move this beyond Postgres when you need to.

I’d keep a small transactional outbox for when you really need it and encourage devs to use it only when absolutely necessary.

I’m currently cleaning up an application that has reached the limit of vertical scaling with Postgres. A significant part of that is because it uses Postgres for every background work queue. Every insert into the queue is in a transaction—do you really want to rollback your change because a notification job couldn’t be enqueued? Probably not. But the ability is there and is so easy to do that it gets overused.

Now I get to go back through hundreds of cases and try to determine whether the transactional insert was intentional or just someone not thinking.

Re: Kafka is Fast – I'll use Postgres

#274

Post by EdwardDiego »

uberduper wrote: post #68
010101010101 wrote: post #41

Earlier quoted context omitted.

> If you don't need what kafka offers, don't use it. This is literally the point the author is making.

It seems like their point was to criticize people for using new tech instead of hacking together unscalable solutions with their preferred database.

Which is crazy, because Kafka is like olllld compared to competing tech like Pulsar and RedPanda. I'm trying to remember what year I started using v0.8, it was probably mid-late 2010s?

Re: Kafka is Fast – I'll use Postgres

#275

Post by asah »

"500 KB/s workload should not use Kafka" - yyyy!!! indeed, I'm running 5MBps logging system through a single node RDS instance costing <$1000/mon (plus 2x for failover). There's easily 4-10x headroom for growth by paying AWS more money and 3-5x+ savings by optimizing the data structure.

Re: Kafka is Fast – I'll use Postgres

#276

Post by EdwardDiego »

loire280 wrote: post #40
uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

In fact, a properly-configured Kafka cluster on minimal hardware will saturate its network link before it hits CPU or disk bottlenecks.

Depends on how you configure the clients, ask me how I know that using a K8s pod id in a consumer group id is a really bad idea - or how setting batch size to 1 and linger to 0 is a really bad idea - the former blows up disk (all those unique consumer groups cause the backing topic to consume a lot of space, as the topic is by default only compacted) and the latter thrashes request handler CPU time.

Re: Kafka is Fast – I'll use Postgres

#277

Post by dagss »

j45 wrote: post #93
johnyzee wrote: post #23

Seems like you would at the very least need a fairly thick application layer on top of Postgres to make it look and act like a messaging system. At that point, seems like you have just built another messaging system. Unless you're a five man shop where everybody just agrees to use that one table, make sure to manage transactions right, cron job retention, YOLO clustering, etc. etc. Performance is probably last on the…

You expose the api on Postgres much like any other group of developers use and call it a day. There’s several implementations of queues to increase the chance of finishing what one is after. https://github.com/dhamaniasad/awesome-postgres

There's a lot of logic involved client side regarding managing read cursors and marking events as processed consumer side. Possibly also client side error queues and so on.

I truly miss a good standard client side library following the Kafka-in-SQL philosophy. I started on in my previous job and we used it internally but it never got good enough that it would be widely used elsewhere, and now I work somewhere else...

(PS: Talking about the pub/sub Kafka-like usecase, not the work queue FOR UPDATE usecase)

Re: Kafka is Fast – I'll use Postgres

#278

Post by hyperbolablabla »

dagss wrote: post #268

I really believe this is the way: Event log tables in SQL. I have been doing it a lot. A downside is the lack of tooling client side. For many using Kafka is worth it simply for the tooling in libraries consumer side. If you just want to write an event handler function there is a lot of boilerplate to manage around it. (Persisting read cursors etc) We introduced a company standard for one service pulling events from…

I for one really dislike Kafka and this looks like a great alternative

Re: Kafka is Fast – I'll use Postgres

#279

Post by EdwardDiego »

asah wrote: post #275

"500 KB/s workload should not use Kafka" - yyyy!!! indeed, I'm running 5MBps logging system through a single node RDS instance costing <$1000/mon (plus 2x for failover). There's easily 4-10x headroom for growth by paying AWS more money and 3-5x+ savings by optimizing the data structure.

I've always said, don't even think about Kafka until you're into MiB/s territory.

It's a complex piece of software that solves a complex problem, but there's many trade-offs, so only use it when you need to.

Re: Kafka is Fast – I'll use Postgres

#280

Post by codeflo »

EdwardDiego wrote: post #271
munchbunny wrote: post #119

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

Do you mean this in the sense that listeners don't remove messages, as one would expect from a queue data structure?
Post reply on HN