Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choose Postgres queue technology

adriano.fyi

Post reply on HN
Print view View original

301–310 of 369 posts

Re: Choose Postgres queue technology

#301

Post by latchkey »

lysecret wrote: post #297
latchkey wrote: post #184

Earlier quoted context omitted.

GCP is really under rated in this regard. Are there any open source implementations of Task Queues? It feels like something that has been missing for years.

Yea, this is the only thing I don't like about them, that I can't test them locally. More generally, is there something like a "on prem cloud" which just replicates say Cloud Tasks (but also other Cloud Apis) using local compute as well as say a local db. For testing / development this would be very cool.

I implemented my tasks as cloud functions, so I just test Tasks the same way I tested functions... by calling the handler function directly.

Re: Choose Postgres queue technology

#302

Post by Lutger »

valenterry wrote: post #263
Lutger wrote: post #254

Earlier quoted context omitted.

This is all true, important and often misunderstood, but beside the point made to which you reply. There's a (sort of) objective trade-off to be made, but another dimension is how familiar you are with the solution and/or how quick can you implement it using documentation and examples. If you happen to know exactly how to create a horizontally scalable microservice based hairball with nodejs, then maybe you are quick…

Then maybe I just don't understand your post. To me it sounds like you say "FAANG-technology" is chosen because of documentation. But I don't think that the documentation of e.g. SQS is better than the postgres (if you can even compare the too). If someone says "I choose X over Y because I used X before (or because X is better documented" then fair enough - but I rarely hear that as an argument when choosing "FAANG-t…

That was exactly what ritzaco said (not me):

> If I want to copy FAANG or another startup, and set up an infinitely scalable queue-based architecture, I can find dozens of high quality guides, tutorials, white papers etc, showing me exactly how to do it.

I'm not sure about this either, though from reading typical developer blogs and listening to the hivemind, you do get the feeling that you must be scalable. Devs often don't really know when (usually not) that becomes important and how far the vast majority of apps can go with monoliths in big boxes (quite far).

Re: Choose Postgres queue technology

#303

Post by chuckhend »

foreigner wrote: post #203

I'm in the market for a Postgres-backed queue system with client libraries in NodeJS _and_ Python. Clients in both languages need to be able to read and write from the queue. Can anybody suggest one?

PGMQ does not require a client library, https://github.com/tembo-io/pgmq so long as your language of choice can run SQL. All the functions live in Postgres, and you just call them with SQL statement. Very similar feel and semantics to SQS.

Re: Choose Postgres queue technology

#304

Post by latchkey »

pjmlp wrote: post #202
latchkey wrote: post #169

Earlier quoted context omitted.

I like how GCP cloud tasks reverses the model. Instead of workers pinging the server asking for work, have the queue ping the worker and the worker is effectively a http endpoint. So you send a message to the server, it queues it and then pings a worker with the message. https://cloud.google.com/tasks/docs/dual-overview

Just like using JMS, MSMQ or similar queues, I fail to see what is so great about it.

JMS is a specification, not an implementation.

Re: Choose Postgres queue technology

#305

Post by jashmatthews »

pqdbr wrote: post #161
phamilton wrote: post #151

Earlier quoted context omitted.

I can share our experience with RabbitMQ/SQS/Sidekiq. Our two major issues have been around the retry mechanism and resource bottlenecks. The key retry problem is "What happens when a worker crashes?". RabbitMQ solves this problem by tying "unacknowledged messages" to a tcp connection. If the connection dies, the in-flight messages are made available to other connections. This is a decent approach, but we hit a lot o…

> Sidekiq solves "What happens when a worker crashes?" by just not solving it. In the free version, those jobs are just lost. I've been using Sidekiq for 11+ years in production and I've never seen this happen. Sidekiq (free version) has a very robust retry workflow. What are you talking about here?

Sidekiq will drop in-progress jobs when a worker crashes. Sidekiq Pro can recover those jobs but with a large delay. Sidekiq is excellent overall but it’s not suitable for processing critical jobs with a low latency guarantee.

https://github.com/sidekiq/sidekiq/wiki/Reliability

Re: Choose Postgres queue technology

#306

Post by phamilton »

imiric wrote: post #171
phamilton wrote: post #151

Earlier quoted context omitted.

I can share our experience with RabbitMQ/SQS/Sidekiq. Our two major issues have been around the retry mechanism and resource bottlenecks. The key retry problem is "What happens when a worker crashes?". RabbitMQ solves this problem by tying "unacknowledged messages" to a tcp connection. If the connection dies, the in-flight messages are made available to other connections. This is a decent approach, but we hit a lot o…

That's very insightful, thanks for sharing. Do you have any experience with NATS, and how would you compare it to RMQ/SQS? The authors claim it guarantees exactly-once delivery with its JetStream component, and it looks very alluring from the documentation, but looks can be deceiving.

> The authors claim it guarantees exactly-once delivery

I find this definition has morphed from one meaningful to developers into one queue implementations like to claim. I've learned this generally means "multiple inserts will be deduped into only one message in the queue".

The only guarantee this `exactly-once` delivery provides is that I won't have two workers given the exact same job. Which is a nice guarantee, but I still have to decide on my processing behavior and am faced with the classic "at most once or at least once" dilemma around partially failed jobs. If I'm building my system to be idempotent so I can safely retry partially failed messages it doesn't do much for me.

Re: Choose Postgres queue technology

#307

Post by cereal_cable »

valenterry wrote: post #263
Lutger wrote: post #254

Earlier quoted context omitted.

This is all true, important and often misunderstood, but beside the point made to which you reply. There's a (sort of) objective trade-off to be made, but another dimension is how familiar you are with the solution and/or how quick can you implement it using documentation and examples. If you happen to know exactly how to create a horizontally scalable microservice based hairball with nodejs, then maybe you are quick…

Then maybe I just don't understand your post. To me it sounds like you say "FAANG-technology" is chosen because of documentation. But I don't think that the documentation of e.g. SQS is better than the postgres (if you can even compare the too). If someone says "I choose X over Y because I used X before (or because X is better documented" then fair enough - but I rarely hear that as an argument when choosing "FAANG-t…

I often find that the tooling I have at work helps speed up the development of more complicated solutions. So saying that FAANG solutions are easy to use and you can be fast at is easy when you have that FAANG support. Even just non-FAANG but large enterprises allow for that, but for startup's is easy to forget how the environment (including tooling) helps speed up all of that work immensely.

So yeah, I find a lot of the more complicated solutions to be simple, but mostly because it's well supported and not by just me.

Re: Choose Postgres queue technology

#308

Post by valenterry »

Lutger wrote: post #302
valenterry wrote: post #263

Earlier quoted context omitted.

Then maybe I just don't understand your post. To me it sounds like you say "FAANG-technology" is chosen because of documentation. But I don't think that the documentation of e.g. SQS is better than the postgres (if you can even compare the too). If someone says "I choose X over Y because I used X before (or because X is better documented" then fair enough - but I rarely hear that as an argument when choosing "FAANG-t…

That was exactly what ritzaco said (not me): > If I want to copy FAANG or another startup, and set up an infinitely scalable queue-based architecture, I can find dozens of high quality guides, tutorials, white papers etc, showing me exactly how to do it. I'm not sure about this either, though from reading typical developer blogs and listening to the hivemind, you do get the feeling that you must be scalable. Devs oft…

Okay, if that is the context then I understand.

But my response would then be that this is a stupid example in the context of this whole submission because that submission talks about postgres and trying to get postgres to scale "infinitely" let alone fulfill other properties like extremely high uptime etc. that is just... insane. No one in their right mind tries to do that with postgres. It is one thing to do queueing with it but "infinitely scalable" is a totally different one.

Therefore I can only say: yeah, to set up "an infinitely scalable queue-based architecture" you should not use postgres and the author in the submission says the same thing.

> Devs often don't really know when (usually not) that becomes important and how far the vast majority of apps can go with monoliths in big boxes (quite far).

Right, they make the wrong trade-offs. That is exactly what I wanted to express with my response.

Re: Choose Postgres queue technology

#309

Post by valenterry »

cereal_cable wrote: post #307
valenterry wrote: post #263

Earlier quoted context omitted.

Then maybe I just don't understand your post. To me it sounds like you say "FAANG-technology" is chosen because of documentation. But I don't think that the documentation of e.g. SQS is better than the postgres (if you can even compare the too). If someone says "I choose X over Y because I used X before (or because X is better documented" then fair enough - but I rarely hear that as an argument when choosing "FAANG-t…

I often find that the tooling I have at work helps speed up the development of more complicated solutions. So saying that FAANG solutions are easy to use and you can be fast at is easy when you have that FAANG support. Even just non-FAANG but large enterprises allow for that, but for startup's is easy to forget how the environment (including tooling) helps speed up all of that work immensely. So yeah, I find a lot of…

But that is not what we are discussing here. From the submission:

> There’s a good chance that you’re already using a relational database, and if that relational database is Postgres, you should consider it for queues before any other software

The point is, if you are already using postgres, then the question is not: should I use postgres for queueing and the rest or should I use postgres for the rest and a FAANG solution for queueing on top of it.

Now the thing is that the FAANG solutions are great in certain ways and allow you to scale a lot and have extremely high availability. But it comes at the cost, for examply those solutions don't support transactions like postgres does. So if you need those (and often you don't know in advance how the business of a startup develops) then now you have to build some technical solution on top of the FAANG solution which is much much slower and more complicated compared to doing it in postgres.

Even if you say that it's more difficult to setup and understand the queueing in postgres (and I agree), I would argue that in the end it is still faster because you don't need to setup and maintain all the infrastructure (yeah, even if it runs in the cloud) unless this is a prototyp and you don't care about security, documentation and all of that and throw it away in the end anyways.

Re: Choose Postgres queue technology

#310

Post by PH95VuimJjqBqy »

ritzaco wrote: post #225

I often see the "engineers copy FAANG infrastructure because they want to be cool, even though their needs are completely different" take as a kind of attack on engineers. But I think a lot of it is also about knowledge and documentation. If I want to copy FAANG or another startup, and set up an infinitely scalable queue-based architecture, I can find dozens of high quality guides, tutorials, white papers etc, showin…

> and that makes the engineer's choice to follow faang look a lot more reasonable.

Your argument is that going with FAANG level designs saves time?

And the crux of your argument is that you're able to find a guide online?

I strongly suspect you don't have a healthy respect for complexity.

Post reply on HN