241–250 of 286 posts
But at scale you probably don't want to manage a bunch of mission critical systems that were jacked into your database server. The database is slow? How do we monitor that?
So I would definitely begin like this, but you need to have a plan to break all of these out sooner or later.
This is exactly how tightly coupled, unmaintainable software is constructed. By picking the tools before understanding the model and building bespoke architecture. You pick the tools that the business model requires. It might be a relational data store. It might not be. You might want an event store. You might want to reduce costs with lambdas and DynamoDB. You may need a pub/sub event broker. The OP clearly loves Po…
> This is exactly how tightly coupled, unmaintainable software is constructed. No. If you're struggling to build software against a DB and then abstract parts to use Redis or ES or whatever in the future, that's kinda a skill issue you or your team have with building poor software to begin with. Nothing to do with using a DB for multiple things like a Queue/Search etc.
It’s the business modeling skill that most developers lack, so they skip it and believe an ERD will magically cover all invariants.
Earlier quoted context omitted.
100% except when Kafka goes wrong, who maintains it?
There are two sizes of companies: those that can afford '1+ dedicated ____-person' and those that can't. Which should filter through to technology choices more than it does.
Earlier quoted context omitted.
It’s also entirely possible that nothing you do in the eventual history of your company hits a scale where this matters.
Sometimes to scale is "continue to satisfy SLAs as service usage increases" and other times to scale is "successfully evolve functional capabilities over time." While the GP may have been referencing the former, embracing "PostgreSQL for Everything" often prohibits the latter.
I think that the "use Postgres for everything" messaging was a necessity, even if it is overstated. Use it until you can demonstrate it doesn't meet your near term needs. When that happens, shift. It wasn't that many years ago when I'd enter situations where people were knee deep in FAANG level infrastructure when postgres on a relatively small instance would have more than been sufficient. I'd suggest they look at converting to postgres to save money & all the energy they spend maintaining their soup. "It won't scale the way we need it!". Sometimes they were demonstrably wrong. Other times they were half-right, in that the real problems was terrible decisions made at the software layer, leading to a situation that required heavier duty infra. Almost never were they actually right* though.
Might they have been right 5 years later? Perhaps. But I know for a fact that none of the ones I encountered were.
Earlier quoted context omitted.
A counter anecdata. We transitioned from a postgres job queue to Rabbit. We had never ending problems after that, many of them were misunderstandings, some where just wrong-fit. We migrated because we had some time on our hands and thought we would alleviate some high pressure jobs. Not only did it not solve the problem, but having written all the code that decides when to pull the next message and what to do with it…
I tried very hard to use postgres as a queue, it was robust but slow once I started to push from more than a few processes/servers. Moving to zeromq initially and sqs after solved all my perf issues, and was still solid.
Earlier quoted context omitted.
I think that’s the way to go. Start with Postgres and only if there are problems, then think about something more specialized. Same for microservices. Start simple and introduce a service when really needed. I hate it when people already start out with 10 or more different systems/services for a few messages per second.
it's not the way to go if you hit the limits very quickly and have to waste immense time migrating.
Earlier quoted context omitted.
Listen to this advice. I had a system that has ~600gb of blob data in bytea that could have easily been an S3 bucket + db reference. It made backups way more of a pain than necessary. It was intentional in the design, because I wanted total consistency with a single backup for the system. It worked great for years. But as we got more and more clients, it really should have been migrated to the above design to make su…
So the original advice actually still stands. You can still start off with Posgres, store it in BYTEA columns, and then work on a plan to use an object storage as you grow. S3 may not be possible and you will be evaluating other options like Minio
But there are still plenty of cases I would say BYTEA is perfectly reasonable choice.
Earlier quoted context omitted.
I think that’s the way to go. Start with Postgres and only if there are problems, then think about something more specialized. Same for microservices. Start simple and introduce a service when really needed. I hate it when people already start out with 10 or more different systems/services for a few messages per second.
I think that using the right tool for the job is important and saves a lot of time in the long run. There are expensive headaches that we have to resolve.
Earlier quoted context omitted.
Doesn't the same argument apply even more to using SQLite instead?
Not really. They are two different paradigms. Use the one that is right for you. SQLite is embedded for local applications with one writer mostly. Postgres is for a client-server architecture with many writers. When you start a project, you generally know which architecture you need.
[1] - https://andersmurphy.com/2025/12/02/100000-tps-over-a-billio...