There is a simpler method. Rabbitmq for accepting new jobs. In the function start a job via spring batch. Basically you get the same effect but using existing tools.
Postgres is a great pub/sub and job server (2019)
131–140 of 209 posts
Re: Postgres is a great pub/sub and job server (2019)
#132Earlier quoted context omitted.
you 100% could, and this thread feels like the twilight zone with how many people are advocating for using a rdbms for (what seems like) most peoples queuing needs.
Why should I rely on yet another microservice when I have PostgreSQL right there?
Everything is a nail, why should I use anything but this hammer?
Re: Postgres is a great pub/sub and job server (2019)
#133I think the key concept here is atomicity. If some API is responsible for creating a job, storing it in the database AND publishing it can never be an atomic operation. Both the database and pub/sub servers are separate network connections from the application server. For example, if you save the record first and then publish, It's quite possibe that you save the record in the database and then lose the connection to…
>If some API is responsible for creating a job, storing it in the database AND publishing it can never be an atomic operation.
We use transactional outbox for that - we insert a record about the event into the event table in the same transaction as the rest of the operation (providing atomicity), and then a special goroutine reads this table and pushes new events to the message broker on a different server. In our design, there are multiple services which might want to subscribe to the event, and the rule is that they shouldn't share their DB's (for proper scaling) so we can't handle event dispatch in some single central app instance. Of course we could implement our own pub/sub server in Go over a DB like Postgres if we wanted, but what's the point of reinventing the wheel if there's already existing battle-tested tools for that, considering you have to reimplement: queues, exchanges, topics, delivery guarantee, proper error handling, monitoring etc.
Re: Postgres is a great pub/sub and job server (2019)
#134I had thought about using postgres as a job queue before, but I couldn't figure out in my head how to make sure two processes didn't both take the same job. The "FOR UPDATE" and "SKIP LOCKED" were the keys to make this work in the article. Essentially, as far as I can tell, "SELECT FOR UPDATE" locks the rows as they're selected (locks are apparently visible outside the transaction), and "SKIP LOCKED" skips over rows…
This. The article seems like "one weird trick that message queue companies HATE" as it's utilizing, as far as I understand, some SQL semantics in a very specific way to cobble together a way of achieving what other software is designed to do out of the box. It seems fine for a toy system, but I wouldn't stake the success of a real company on this approach. One could also use DNS TXT as an RDBMS with some interesting…
If you want to carry messages across the internet SQS et al is fine. But within the same system, e.g. in the same computer, or cluster, it makes much more sense to use something like this rather than something like SQS. Different tool, different job.
Re: Postgres is a great pub/sub and job server (2019)
#135Earlier quoted context omitted.
Such a server is 400$/mo, a backend developer that can confidently maintain kafka in production is significantly more expensive!
It's that much on a popular cloud platform, you can buy this for 3-4 times that amount and use it for years.
Or rent it for a lot less at a traditional hosting company.
Re: Postgres is a great pub/sub and job server (2019)
#136Earlier quoted context omitted.
well most cloud queues do not support priorities you can only create multiple subscriptions and prefer the messages from the higher one. so in the end you would built a system on a system anyway. also these queues lock you in quite hardly (and do not work on premise) Edit: also keep in mind most queues do not like "slow consumers" i.e. if your workload is bursty with long processing times, a database might be a bette…
>well most cloud queues do not support priorities you can only create multiple subscriptions At least on GCP PubSub, a subscription is a separate concept from a topic/queue. If you want different priorities, you create multiple topics. You create multiple subscriptions when you want to fan out a single message to multiple workers. As far as I know, multiple subscriptions have nothing to do with priorities. Can you ex…
ah yeah topics...
I basically meant topics.
But having multiple topics for priority is still way harder than lets say the rabbitmq priority stuff or the postgres stuff.
Re: Postgres is a great pub/sub and job server (2019)
#137Earlier quoted context omitted.
What is your idea of 'most cases'? I've personally written real-time back-of-house order-tracking with rails and postgres pubsub (no redis!), and wrote a record synchronization queuing system with a table and some clever lock semantics that has been running in production for several years now -- which marketing relies upon as it oversees 10+ figures of yearly topline revenue. Neither of those projects were FAANG scal…
>some clever lock semantics Most senior+ engineers that I know would hear that and recoil. Getting "clever" with concurrency handling in your home-rolled queuing system is not something that coworkers, especially more senior coworkers, will appreciate inheriting, adapting, and maintaining. Believe me. I get that you're trying to flex some cool thing that you built, but it doesn't really have any bearing on the concep…
> Most senior+ engineers that I know would hear that and recoil. Getting "clever" with concurrency handling in your home-rolled queuing system is not something that coworkers, especially more senior coworkers, will appreciate inheriting, adapting, and maintaining. Believe me.
I am both a "senior+ engineer" that has inherited such systems and an author of such systems. I think you're overreacting.
Concurrency Control (i.e., "lock semantics") exists for a reason: correctness. Using it for its designed purpose is not horror. Yes, like any tool, you need to use it correctly. But you don't just throw away correctness because you don't want to learn how to use the right tool properly.
I have inherited poorly designed concurrency systems (in the database); yes, I recoiled in horror and did not appreciate it. So you know what I did? I fixed the design, and documented it to show others how to do it correctly.
I have also inherited OOB "Queuing Systems" that could not possibly be correct because they weren't integrated into the DB's built-in and already-used correctness system: Transactions and Concurrency Control. Those were always more horrific than poorly-implemeneted in-DB solutions. Integrating two disparate stores is always more trouble than just fixing one single source.
----
> I get that you're trying to flex some cool thing that you built, but it doesn't really have any bearing on the concept of "most cases" because it's an anecdote. Queuing systems are a thing for a reason, and in most cases, using them makes more sense than writing your own.
I get that you're trying to flex that you use turnkey Queueing Systems, but it doesn't really have any bearing on the concept of "most cases", because all you've presented are assertions without backing. Queuing systems are good, for a specific kind of job, but when you need relational logic you better use one that supports it. And despite what MongoDB and the NoSQL crowd has been screaming hoarsely for the past decade, in most cases, you have relational logic.
Re: Postgres is a great pub/sub and job server (2019)
#138Earlier quoted context omitted.
Why should I rely on yet another microservice when I have PostgreSQL right there?
Everything is a nail, why should I use anything but this hammer?
Postgres happens to be a very good hammer, thank you very much. You should try it sometime.
But seriously though, postgres's relational logic implementation makes for a very good queueing system for most cases. It's not a hack that's bolted on top. I know that's how quite a few "DBs" are designed and implemented, and maybe you've been burned by too many of them, but Postgres is solid. I've seen it inside and out.
Re: Postgres is a great pub/sub and job server (2019)
#139Earlier quoted context omitted.
What is your idea of 'most cases'? I've personally written real-time back-of-house order-tracking with rails and postgres pubsub (no redis!), and wrote a record synchronization queuing system with a table and some clever lock semantics that has been running in production for several years now -- which marketing relies upon as it oversees 10+ figures of yearly topline revenue. Neither of those projects were FAANG scal…
>some clever lock semantics Most senior+ engineers that I know would hear that and recoil. Getting "clever" with concurrency handling in your home-rolled queuing system is not something that coworkers, especially more senior coworkers, will appreciate inheriting, adapting, and maintaining. Believe me. I get that you're trying to flex some cool thing that you built, but it doesn't really have any bearing on the concep…
Well, you'd have to see it before you judge. It's super simple, like 5 or 10 lines total. Handles 1000x+ the traffic it sees. In any case concurrency is nothing to be afraid of. Do they not teach dining philosophers any more?
My point is that postgres is a swiss army knife and you and anyone else would be remiss to not fully understand what it is capable of and what you can do with it. Entire classes of software baggage can be eliminated for "most" use cases. One could even argue that reaching for all these extra fancy specialized tools is a premature optimization. Plus, who could possibly argue against having fewer moving parts?
Re: Postgres is a great pub/sub and job server (2019)
#140Earlier quoted context omitted.
Why should I rely on yet another microservice when I have PostgreSQL right there?
Everything is a nail, why should I use anything but this hammer?
Make every system as complex as you can with tech you are not really familiar with is a good plan for your small team? Under a 100 people, your company does not have 100 devops etc to make sure all these 'best of breed' tools actually managed properly in production. If a service on top of postgres dies, I will find out why very quickly; on Kafka, even though I have used it a bunch of times, I usually have no clue; just restart and pray. Why would I force myself to use another tool when postgres actually works well enough? Resume driven?
Sometimes I agree with best tool for the job; if the constraints make something a very clear winner; if the difference is marginal for the particular case at hand, I pick what I/we know (I would actually argue that IS the best tool for the job; but in absolute 'what could happen in the future' terms it probably is not).