Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres is a great pub/sub and job server (2019)

webapp.io

Post reply on HN
Print view View original

111–120 of 209 posts

Re: Postgres is a great pub/sub and job server (2019)

#111

Post by option_greek »

daenz wrote: post #84
KptMarchewa wrote: post #81

Earlier quoted context omitted.

Which complexity? Of running a SQL query on the same database you're already using, vs writing code to support some other new system?

>Of running a SQL query on the same database you're already using Go back and read OPs link. They create new SQL types, tables, triggers, and functions, with non-trivial and very unforgiving atomic logic. And every system that needs to read or write from this "db queue" needs to leverage specific queries. That's the complexity. >vs writing code to support some other new system You mean using a stable well maintained…

Not necessarily. Many languages like Java already have queuing libraries that operate on rdbms through JPA. So not even a single additional line needs to be written for this to work. We got ours working in a day and it works great. I don't know if other languages have these libraries but I'm inclined to believe they do (at least nodejs has).

Re: Postgres is a great pub/sub and job server (2019)

#112

Post by fnord77 »

staticassertion wrote: post #32
petilon wrote: post #10

Just because something can be used to do something doesn't mean it should. Kafka is specifically designed for this purpose, it is free, and it is easy to learn and use. If "starting with Postgres and then switching out when the time comes" saves money then I can understand. Otherwise use the right tool for the right job, right from the start.

Kafka is not a queue. Kafka's parallelism is limited by the number of partitions you allocate, and you have to be sure to avoid head of line blocking. Not the case with a queue.

what is "head of line" blocking?

Re: Postgres is a great pub/sub and job server (2019)

#113

Post by nodesocket »

Anybody using graphile-worker[1] in production/heavy load? It looks awesome, and I coded up some simple prototype tasks (email, sms, etc), but question how it truly scales. They claim horizontal scaling is trivial.

> graphile-worker is horizontally scalable. Each instance has a customisable worker pool, this pool defaults to size 1 (only one job at a time on this worker) but depending on the nature of your tasks (i.e. assuming they're not compute-heavy) you will likely want to set this higher to benefit from Node.js' concurrency.

[1] https://github.com/graphile/worker

Re: Postgres is a great pub/sub and job server (2019)

#114

Post by option_greek »

peterhunt wrote: post #29
paxys wrote: post #13

Earlier quoted context omitted.

What does scale have to do with it? Pub/sub as an architectural pattern could be equally relevant for your use case whether there are a hundred users in your system or a billion. And Kafka isn't the only solution for it. There are many lightweight pub/sub and queuing systems which also don't involve needlessly adding abstraction layers and application code into an RDBMS.

I think massive scale is the only reason you'd really want to adopt something like Kafka. If you're 10k inserts/s or less then there's no reason not to do everything in a single big relational DB where you get the warm fuzzy feeling of transactions, point in time backups, scalable read replication, etc

Yes but then 'enterprise architects' can't put their shiny cloud certifications to use. They need pubsub regardless of the scale because that's what is recommended by big cloud. And of course the companies also don't mind because there is no immediate devops cost for it. The problems like complexity of debugging, lockin won't hit home at the time of design.

Re: Postgres is a great pub/sub and job server (2019)

#116

Post by daenz »

option_greek wrote: post #111
daenz wrote: post #84

Earlier quoted context omitted.

>Of running a SQL query on the same database you're already using Go back and read OPs link. They create new SQL types, tables, triggers, and functions, with non-trivial and very unforgiving atomic logic. And every system that needs to read or write from this "db queue" needs to leverage specific queries. That's the complexity. >vs writing code to support some other new system You mean using a stable well maintained…

Not necessarily. Many languages like Java already have queuing libraries that operate on rdbms through JPA. So not even a single additional line needs to be written for this to work. We got ours working in a day and it works great. I don't know if other languages have these libraries but I'm inclined to believe they do (at least nodejs has).

Do you have to set up the triggers, tables, and procedures beforehand or how does that work?

Re: Postgres is a great pub/sub and job server (2019)

#117

Post by throwaway984393 »

Webapp.io people: I really don't get how your service works, exactly. I see the broad strokes, but I don't know how it will fit my own use cases, because I cannot see how it would integrate with my architecture or workflows. Please create some more diagrams with actual examples of all the use cases you hint at on your website but don't actually show examples of (other than your config file examples, which do not answer any of my questions). Even if I could use your thing at work, I won't even try now, because I can't see how it would apply, and I don't want to spend an hour digging into your tech docs to figure out if it would apply.

Re: Postgres is a great pub/sub and job server (2019)

#118

Post by stillicidious »

jbverschoor wrote: post #73
SahAssar wrote: post #55

Earlier quoted context omitted.

I assume that is their main database for everything, not just for pub/sub. One of the big benefits of doing it that way is that you have proper transaction handling across jobs and their related data.

Come on man… you can run the whole thing off if a few Gb instance. Such a huge instance should be able to do about 100k a second!

Be careful not to confuse average load with peak instantaneous load. Bursty workloads are the bane of capacity planners everywhere.

Re: Postgres is a great pub/sub and job server (2019)

#119

Post by daenz »

tomc1985 wrote: post #105
daenz wrote: post #63

Earlier quoted context omitted.

you 100% could, and this thread feels like the twilight zone with how many people are advocating for using a rdbms for (what seems like) most peoples queuing needs.

Dude you are seriously underestimating postgres' versatility. It does so many different things, and well!

I'm not underestimating anything. I am advocating for the right tool for the job. I have a hard time believing, despite the skewed sample size in this thread, that most people think using postgres as a message queue for most cases makes the most sense.

Re: Postgres is a great pub/sub and job server (2019)

#120

Post by dirkt »

colinchartier wrote: post #19

Author here! A few updates since this was published two years ago: - The service mentioned (now called https://webapp.io ) eventually made it into YC (S20) and still uses postgres as its pub/sub implementation, doing hundreds of thousands of messages per day. The postgres instance now runs on 32 cores and 128gb of memory and has scaled well. - We bolstered Postgres's PUBLISH with Redis pub/sub for high traffic code p…

What are the options to use Postgres pub/sub with Java? Because the usual Java libraries don't seem to support the pub/sub functionality well, you have to actively poll when you want to subscribe.
Post reply on HN