How about metrics? Would it be possible to do this in Postgres as well? I'm talking e.g. about simple counters that are not directly vital to business logic. I'm currently using Redis for those things as I'm thinking that Postgres seems to be the wrong choice for this kind of data, would anyone disagree?
PostgreSQL is enough
51–60 of 323 posts
Re: PostgreSQL is enough
#52Postgres is enough as long as you have a good multi-tenant setup e.g. a separate database per customer. Ran a single postgres instance with multi-tenant SaaS product that crossed 4B records in a few tables, even with partitions and all the optimization in the world, it still hurts to have one massive database. We still got bought tho, so I will agree its enough
> Postgres is enough as long as you have a good multi-tenant setup e.g. a separate database per customer.
Separate per customer has a lot of advantages (especially around customer security and things like deletion) but you can also shard by customer right from the beginning; customer #1 in the "odd" shard, customer #2 in the "even" shard etc. Switching to database per customer if that is working well is relatively easy so you're future proofed both ways.
Re: PostgreSQL is enough
#53Re: PostgreSQL is enough
#54Thank you for this post. I've been looking high and low for weeks trying to find a way to connect Looker to a GraphQL API unsuccessfully and apparently "PostgreSQL emulators" are called FDWs as I learned from this link. I had searched every term I could think of for how to simulate or proxy a SQL database into an API but calling it a foreign data wrapper would have never occurred to me.
Did you try describing the problem and the thing you're looking for to chatgpt to see if it knows of a name? I find that making connections like this is one of the things it actually does well
Remember when you could do this with Google?
Re: PostgreSQL is enough
#55How about metrics? Would it be possible to do this in Postgres as well? I'm talking e.g. about simple counters that are not directly vital to business logic. I'm currently using Redis for those things as I'm thinking that Postgres seems to be the wrong choice for this kind of data, would anyone disagree?
So, having worked on a managed postgres service, we had an extension pushing metrics from the database, but we also used redis to keep service metrics
Redis is great. This gist is more about how it's up to you how much you centralize on postgres. But overall being able to offload from the database has value, so "PostgreSQL is Enough" should not read as "You shouldn't need more than PostgreSQL"
Metrics in an unlogged table could be great if you want to query those metrics against existing data. It depends
Re: PostgreSQL is enough
#56I often go down rabbit holes like this, trying to collapse and simplify the application stack. But inevitably, as an application grows in complexity, you start to realize _why_ there's a stack, rather than just a single technology to rule them all. Trying to cram everything into Postgres (or lambdas, or S3, or firebase, or whatever other tech you're trying to consolidate on) starts to get really uncomfortable. That s…
I think SQS is cheap enough to build on as a messaging queue even if you're not hosting within AWS. Out of the widely underrated AWS services include SNS and SES and they are not a bad choice even if you're not using AWS for compute and storage.
The problem is rarely cost, it's operational overhead.
Using SQS for a queue rather than my already-existing Postgres means that I have to:
- Write a whole bunch of IaC, figuring out the correct access policies - Set up monitoring: figure out how to monitor, write some more IaC - Worry about access control: I just increased the attack surface of my application - Wire it up in my application so that I can connect to SQS - Understand how SQS works, how to use its API
It's often worth it, but adding an additional moving piece into your infra is always a lot of added cognitive load.
Re: PostgreSQL is enough
#57This makes a strong case, but I've decided to start every new project with sqlite and not switch until absolutely necessary. If Postgres is the 90% case, then sqlite is the 80% case and is also dead simple to get going and genuinely performant. So when vertical scaling finally fails me, I know I'll be at a wonderful place with what I'm building.
> [...] sqlite is the 80% case and is also dead simple to get going and genuinely performant. I don't understand this. PostgreSQL is ALSO dead simple to get going, either locally or in production. Why not just start off at 90%? I mean, I get there are a lot of use cases where sqlite is the better choice (and I've used sqlite multiple times over the years, including in my most recent gig), but why in general?
I think "dead simple" is not doing anyone any favors when it is being used to try to equate the simplicity of things.
It's obviously a lot simpler to just have a file, than to have a server that needs to be connected to, as long as we're still talking about running things on regular computers.
Re: PostgreSQL is enough
#58I often go down rabbit holes like this, trying to collapse and simplify the application stack. But inevitably, as an application grows in complexity, you start to realize _why_ there's a stack, rather than just a single technology to rule them all. Trying to cram everything into Postgres (or lambdas, or S3, or firebase, or whatever other tech you're trying to consolidate on) starts to get really uncomfortable. That s…
PG works really well as a message queue and there's several excellent implementations on top of it.
Most systems are still going to need Redis involved just as a coordinator for other pub/sub related work unless you're using a stack that can handle it some other way (looking at BEAM here).
But there are always going to be scenarios as an application grows where you'll find a need to scale specific pieces. Otherwise though, PostgreSQL by itself can get you very, very far.
Re: PostgreSQL is enough
#59Earlier quoted context omitted.
SQS is at least once PG can give you exactly once
Not sure why this is making people upset.
Because it's incorrect. If you have any non-postgres side-effect, you can't have exactly-once (unless you do 2PC or something like that). There isn't any technology that gives you "exactly once" in the general case.
Re: PostgreSQL is enough
#60This makes a strong case, but I've decided to start every new project with sqlite and not switch until absolutely necessary. If Postgres is the 90% case, then sqlite is the 80% case and is also dead simple to get going and genuinely performant. So when vertical scaling finally fails me, I know I'll be at a wonderful place with what I'm building.
> [...] sqlite is the 80% case and is also dead simple to get going and genuinely performant. I don't understand this. PostgreSQL is ALSO dead simple to get going, either locally or in production. Why not just start off at 90%? I mean, I get there are a lot of use cases where sqlite is the better choice (and I've used sqlite multiple times over the years, including in my most recent gig), but why in general?
But PostgreSQL is not dead simple when compared to SQLite.