Live data from Hacker News

How to use Postgres for everything

github.com

71–80 of 181 posts

Re: How to use Postgres for everything

#71
This idea that Postgres should be used for everything really need to die in a professional context.

I was appointed in a company of 10 dev that did just that. All backend code was PostgreSQL functions, event queue was using Postgres, security was done with rls, frontend was using posgtraphile using graphql to expose these functions, triggers were being used to validate information on insert/update.

It was a mess. Postgres is a wonderful database, use it as a database. But don't do anything else with it.

Before some people come and say "things were not done the right way, people didn't know what they were doing". The dev were all fan of Postgres contributing to the projects around, there was a big review culture so people were really trying to the best.

The queue system was locking all the time between concurrent requests => so queue system with postgres works for a pet project

All the requests were 3 or 4 times longer due to fact that you have to check the rls on each row. We have also all pour API migrated now and each time the sql duration decrease by that factor ( and it is the exact same sql request ). And the db was locking all the time because of that as it feels likes rls breaks the deadlock detection Postgres algorithm

SQL is super verbose a language, you spend your time repeating the same line of code , it makes basic function about 100 lines long when they are 4-5 lines in nodes js

It is impossible to log things inside these functions to have to make sure things will work and if it doesn't you have no way to know where the code did go through

You can't make external API call, so you have to use a queue system to make any basic things there

There are not real lib , so everything need to be reimplemeted

It is absolutely not performant to code inside the db, you can't do a map so you O(n2) code all the time

API were needed for the external world , so there was actually another service in front of the database for some case and a lot of logic were reimplemeted inside it

There was a downtime at each deployment as we had to remove all the rls and recreate them ( despite the fact that all code was in insert if not update clauses) it worked at the beginning but at some point in time it stopped working and there was no way to find why, so drop all rls and recreate them

It is impossible to hire dev that wants to work on that stack and be business oriented , you would attract only purely tech people that care only about doing there own technical stuff

We are almost out of it now after 1 year of migration work and I don't see anything positive about this Postgres do everything culture compared to a regular node js + Postgres as a database + sqs stack

So to conclude, as a pet project it can be great to use Postgres like that, in a professional context you are going to kill the company with this technical choice

Re: How to use Postgres for everything

#73

Just don't use a single Postgres DB for everything as you scale up to 100+ engineers. You'll inevitably get database-as-the-API. Now if you have the actual technical leadership [1] to scale your systems by drawing logical and physical boundaries so that each unit has its own Postgres? Yeah Postgres for everything is solid. [1] Surprisingly rare I've found. Lots of "successful" CTOs who don't do this hard part.

Database-as-the-API can scale surprisingly far, particularly if you sell a single-tenant shard to each customer and therefore a separate database to each customer. Drawing logical software boundaries before Product even knows what the domain looks like (i.e. which features will sell) is quite risky.

Re: How to use Postgres for everything

#74

While we are it - are there any good resources on how to best self host a Postgres database? Any tips and tricks, best practices, docker / no docker etc? I’m looking to self host a database server for my multiple pet projects, but I would love to get backups, optimizations and other stuff done well.

afaik:

On promise: Use containers but the data folder should be mounted volume

On cloud/k8s: Just use a managed DB, setting up a DB in k8s is hard because the filesystem

Re: How to use Postgres for everything

#75

While we are it - are there any good resources on how to best self host a Postgres database? Any tips and tricks, best practices, docker / no docker etc? I’m looking to self host a database server for my multiple pet projects, but I would love to get backups, optimizations and other stuff done well.

Not the answer you were looking for, but I had been shopping recently for slightly overlapping reasons (I want to ship projects, but they are all smaller MVPs that might run for a while and I didn't want to pay for 1 database service for each one on Render).

I found https://www.thenile.dev/pricing which supports which apparently supports unlimited databases.

Re: How to use Postgres for everything

#76

Having just spent the better part of two weeks integrating Apache Age for Graph data, just to realize the project is stale and a mess, don’t take this list on face value. Now hoping for better results with DGraph, but it seems that graph databases are living a precarious existence.

I wonder what's the catch with Dgraph? Why not chose it above Neo4j? I'm asking because the graph db projects I've been involved in has all used Neo4j and it would be nice to know of a good alternative.

I want it to be as free as possible, neo4j only let’s you run a single database in non-Enterprise mode. We are building a consumer product, where the database is embedded and not centralized in a cloud, so my focus is perhaps different from most. Both neo4j and dgraph comes with additional non-compete clauses, but DGraph’s work for our use case.

Subjectively I’d prefer neo4j. I have been following the company since its inception.

Re: How to use Postgres for everything

#77

Having just spent the better part of two weeks integrating Apache Age for Graph data, just to realize the project is stale and a mess, don’t take this list on face value. Now hoping for better results with DGraph, but it seems that graph databases are living a precarious existence.

Haaaa same here for apache age. Can you elaborate a little please

The original sponsor of the project just withdrew all resources, the state of the existing codebase is far from mature, the client I tried (python) was really shaky and the lidt goes on. As for the precarious life of graph database companies. DGraph also went though being sold recently, and OrientDB that I also liked, as acquired by SAP, only to be abandoned. Neo4j has stood its time, but the licensing doesn’t fit our needs.

Re: How to use Postgres for everything

#78
post #70

PGQueuer is a lightweight job queue for Python, built entirely on PostgreSQL. It uses SKIP LOCKED for efficient and safe job processing, with a minimalist design that keeps things simple and performant. If you’re already using Postgres and want a Python-native way to manage background jobs without adding extra infrastructure, PGQueuer might be worth a look: GitHub - https://github.com/janbjorge/pgqueuer

I always wondered about the claim that SKIP LOCKED is all that efficient. Surely there are lots of cases where this is a really suboptimal pattern.

Simple example: if you have a mixture of very short jobs and longer duration jobs, then there might be hundreds or thousands of short jobs executed for each longer job. In such a case the rows in the jobs table for the longer jobs will be skipped over hundreds of times. The more long-running jobs running concurrently, the more wasted work as locked rows get skipped again and again. It wouldn't be a huge issue if load is low, but surely a case where rows get moved to a separate "running" table would be more efficient. I can think of several other scenarios where SKIP LOCKED would lead to lots of wasted work.

Re: How to use Postgres for everything

#79

Earlier quoted context omitted.

If you don't have any discipline it becomes hell. Not to mention that a random team writing a migration that locks a key shared table (or otherwise chokes resources) now causes outages for everyone .

Right but in this "100-engineer" scenario you'd have hoped the following would have happened: - Docs and guidelines on migrations would have been written - Some level of approval and review is required before execution These are things that isn't really postgres specific, any company that doesn't have those is going to be a nightmare.

If teams have technical boundaries defined at a higher level in the stack (e.g. APIs) and so they don't share a database, you don't need loads of process and docs and architectural meetings to coordinate. Letting teams delivery independently is a good architectural feature.

Re: How to use Postgres for everything

#80

Earlier quoted context omitted.

> You'll inevitably get database-as-the-API I think that's actually the point of "postgres for everything"?

If you don't have any discipline it becomes hell. Not to mention that a random team writing a migration that locks a key shared table (or otherwise chokes resources) now causes outages for everyone .

Not a problem until it's a problem.
Post reply on HN