Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

191–200 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#191

Post by nyrikki »

> The claim isn’t that Postgres is functionally equivalent to any of these specialized systems. The claim is that it handles 80%+ of their use cases with 20% of the development effort. (Pareto Principle)

Lots of us that built systems when SQL was the only option, know that doesn’t hold overtime.

SStable backed systems have their applications, and I have never seen dedicated Kafka teams like we used to have with DBAs

We have the tools to make decisions based on real tradeoffs.

I highly recommend people dig into the appropriate tools to select vs making pre-selected products fit an unknown problem domain.

Tools are tactics, not strategies, tactics should be changeable with the strategic needs.

Re: Kafka is Fast – I'll use Postgres

#192

Post by zX41ZdbW »

PeterCorless wrote: post #103
kragen wrote: post #80

Earlier quoted context omitted.

This sounded interesting to me, and it looks like the plan is to make Redpanda open-source at some point in the future, but there's no timeline: https://github.com/redpanda-data/redpanda/tree/dev/licenses

Correct. Redpanda is source-available. When you have C++ code, the number of external folks who want to — and who can effectively, actively contribute to the code — drops considerably. Our "cousins in code," ScyllaDB last year announced they were moving to source-available because of the lack of OSS contributors: > Moreover, we have been the single significant contributor of the source code. Our ecosystem tools have…

The statement is untrue. For example, ClickHouse is in C++, and it has thousands of contributors with hundreds of external contributors every month.

Re: Kafka is Fast – I'll use Postgres

#193

Post by munchbunny »

loftsy wrote: post #46

I am about to start a project. I know I want an event sourced architecture. That is, the system is designed around a queue, all actors push/pull into the queue. This article gives me some pause. Performance isn't a big deal for me. I had assumed that Kafka would give me things like decoupling, retry, dead-lettering, logging, schema validation, schema versioning, exactly once processing. I like Postgres, and obviously…

> I had assumed that Kafka would give me things like decoupling, retry, dead-lettering, logging, schema validation, schema versioning, exactly once processing.

If you don't need a lot of perf but you place a premium on ergonomics and correctness, this sounds more like you want a workflow engine? https://github.com/meirwah/awesome-workflow-engines

Re: Kafka is Fast – I'll use Postgres

#194

Post by jimbokun »

Jupe wrote: post #34
jimbokun wrote: post #16

For me the killer feature of Kafka was the ability to set the offset independently for each consumer. In my company most of our topics need to be consumed by more than one application/team, so this feature is a must have. Also, the ability to move the offset backwards or forwards programmatically has been a life saver many times. Does Postgres support this functionality for their queues?

Isn't it just a matter of having each consumer use their own offset? I mean if the queue table is sequentially or time-indexed, the consumer just provides a smaller/earlier key to accomplish the offset? (Maybe I'm missing something here?)

Yes.

Is a queuing system baked into Postgres? Or there client libraries that make it look like one?

And do these abstractions allow for arbitrarily moving the offset for each consumer independently?

If you're writing your own queuing system using pg for persistence obviously you can architect it however you want.

Re: Kafka is Fast – I'll use Postgres

#195

Post by Sparkyte »

joaohaas wrote: post #168
Sparkyte wrote: post #167

You can also use Redis as a queue if the data isn't in danger of being too important.

Even if the data is important, you can enable WAL and make sure the worker/consumer gets items by RPOPLPUSHing to a working queue. This way you can easily requeue the data if the worker ever goes offline mid-process.

Very true.

Re: Kafka is Fast – I'll use Postgres

#196

Post by jimbokun »

altcognito wrote: post #47
jimbokun wrote: post #16

For me the killer feature of Kafka was the ability to set the offset independently for each consumer. In my company most of our topics need to be consumed by more than one application/team, so this feature is a must have. Also, the ability to move the offset backwards or forwards programmatically has been a life saver many times. Does Postgres support this functionality for their queues?

The article basically states unless you need a lot of throughput, you probably don't need Kafka. (my interpretation extends to say) You probably don't need offsets because you don't need multi-threaded support because you don't need multiple threads. I don't know what kind of native support PG has for queue management, the assumption here is that a basic "kill the task as you see it" is usually good enough and the si…

Well in my workplace we need all of those things.

Re: Kafka is Fast – I'll use Postgres

#198

Post by loftsy »

munchbunny wrote: post #193
loftsy wrote: post #46

I am about to start a project. I know I want an event sourced architecture. That is, the system is designed around a queue, all actors push/pull into the queue. This article gives me some pause. Performance isn't a big deal for me. I had assumed that Kafka would give me things like decoupling, retry, dead-lettering, logging, schema validation, schema versioning, exactly once processing. I like Postgres, and obviously…

> I had assumed that Kafka would give me things like decoupling, retry, dead-lettering, logging, schema validation, schema versioning, exactly once processing. If you don't need a lot of perf but you place a premium on ergonomics and correctness, this sounds more like you want a workflow engine? https://github.com/meirwah/awesome-workflow-engines

Perhaps I do. I know that I don't want a system defined as a graph in yaml. Or no code. These options are over engineered for my use case. I'm pretty comfortable building some docker containers and operating them and this is the approach I want to use.

I'm checking out the list.

Re: Kafka is Fast – I'll use Postgres

#199

Post by loftsy »

singron wrote: post #113
loftsy wrote: post #46

I am about to start a project. I know I want an event sourced architecture. That is, the system is designed around a queue, all actors push/pull into the queue. This article gives me some pause. Performance isn't a big deal for me. I had assumed that Kafka would give me things like decoupling, retry, dead-lettering, logging, schema validation, schema versioning, exactly once processing. I like Postgres, and obviously…

Kafka also doesn't give you all those things. E.g. there is no automatic dead-lettering, so a consumer that throws an exception will endlessly retry and block all progress on that partition. Kafka only stores bytes, so schema is up to you. Exactly-once is good, but there are some caveats (you have to use kafka transactions, which are significantly different than normal operation, and any external system may observe a…

Very interesting.

I need a durable queue but not indefinitely. Max a couple of hours.

What I want is Google PubSub but open source so I can self host.

Re: Kafka is Fast – I'll use Postgres

#200

Post by loftsy »

mrkeen wrote: post #109
loftsy wrote: post #46

I am about to start a project. I know I want an event sourced architecture. That is, the system is designed around a queue, all actors push/pull into the queue. This article gives me some pause. Performance isn't a big deal for me. I had assumed that Kafka would give me things like decoupling, retry, dead-lettering, logging, schema validation, schema versioning, exactly once processing. I like Postgres, and obviously…

Event-sourcing != queue. Event-sourcing is when you buy something and get a receipt, you go stick it in a shoe-box for tax time. A queue is you get given receipts, and you look at them in the correct order before throwing each one away.

True.

I think my system is sort of both. I want to put some events in a queue for a finite set of time, process them as a single consolidated set, and then drop them all from the queue.

Post reply on HN