Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do you need separate systems when you already have Postgres?

postgresisenough.dev

Post reply on HN
Print view View original

81–90 of 90 posts

Re: Do you need separate systems when you already have Postgres?

#81

Post by JsonDemWitOster »

PaulHoule wrote: post #41
JsonDemWitOster wrote: post #31

I'll do you one better: do you really need Postgres and all these extensions when you already have a filesystem? > only after pushing Postgres to its limits, documenting why it was insufficient, and accepting the operational cost of the alternative I love Postgres as a DB but, really, this is ridiculous. No doubt these extensions can do the job well-enough but you might as well invest in learning the right tool for t…

Filesystems aren't efficient for small bits of data. Like right now I am thinking about a system that has a password reset process and you need to keep track of a user id and a reset token, one or two timestamps, maybe a state variable and a flag or two. That's well under 100 bytes and the cluster size for a typical fs is 4kb or more plus there is the cost of the directory entry. If the OS is Windows it has to ask th…

I can no longer edit my comment above so I'm just leaving this here, picking you arbitrarily out of the handful that addressed my filesystem suggestion:

I am being sarcastic to show what a ridiculous idea it is to shoehorn everything into Postgres. If you find eschewing a relational DB for the filesystem incredulous, you should maybe reconsider if eschewing specialized software in favor of Postgres is advisable.

Re: Do you need separate systems when you already have Postgres?

#82

Post by JsonDemWitOster »

pooloo wrote: post #47
JsonDemWitOster wrote: post #31

I'll do you one better: do you really need Postgres and all these extensions when you already have a filesystem? > only after pushing Postgres to its limits, documenting why it was insufficient, and accepting the operational cost of the alternative I love Postgres as a DB but, really, this is ridiculous. No doubt these extensions can do the job well-enough but you might as well invest in learning the right tool for t…

I don't understand why the answer is to always bloat the system with more specialized software and technical debt, instead of optimizing the existing system. If Postgres can truly handle all of these situations, then mastery of that one tool should be focused on. I guess its more the rapid start-up mindset to get it up and running fast to sell the company, and leave the problem for someone else which is why a lot of…

As I don't want to repeat myself, with emphasis added:

> No Silver Bullet, No Free Lunch, and all that. If your problem domain really warrants something outside of relational storage, you're gonna pay that complexity cost one way or another. You can't escape it by shoehorning everything in Postgres, fantastic as a DB as it is.

So as not to bloat the system, you will end up bloating your Postgres installation instead.

Also, Unix principle/KISS: do one thing and only one thing well. Of course this comes back down to your definition of "one thing". "_Any_ Data Storage" seems to be your one thing but not for me. "Relational Data Storage" is a well-scoped one thing for me.

I'm not saying don't take advantage of the plugin system. But we must exercise good sense so as not to abuse it when we avail of its advantages. TFA's suggestion of "only after pushing Postgres to its limits, documenting why it was insufficient, and accepting the operational cost of the alternative" does not sound like good sense. It sounds like a personal and an organizational burnout waiting to happen. Wait until PG is at its limit does not sound like technical debt to you?

> If Postgres can truly handle all of these situations, then mastery of that one tool should be focused on.

Thus you have conceived the conundrum of The Database of Theseus: after how many plugins is your Postgres cluster no longer a mere Postgres cluster to the point where it is unreasonable for a seasoned/certified Postgres DBA to have a good grasp of the system in a reasonable time span?

This is almost textbook https://en.wikipedia.org/wiki/Inner-platform_effect in that this is the second paragraph of the "Examples" section. I acknowledge that Postgres' plugin system mitigates this somewhat but I just can't be convinced that all those plugins will play well with each other for all eternity and especially under stress (unless you make a PG instance for each sub-system, in which case, refer to my original comment).

For the record

- I've used PG for caching with UNLOGGED. Still had Redis for a message queue and other caching. - I've used PostGIS because it made sense to be able to make geo query when latlon is already a (small) part of your schema. - I've used JSONB but only because the alternative is Mongo. I still had an Elasticsearch cluster indexing all that data for search.

Re: Do you need separate systems when you already have Postgres?

#83

Post by JsonDemWitOster »

h1fra wrote: post #8

I have yet to have used UNLOGGED table in production, but I really want to try at some point. Anybody actually replaced a large Redis instance with this?

I have but only to cache data used by cron/bg jobs. We still had Redis for an actual platform system cache. The thing is that Redis cache was created and managed by some framework middleware so it did not seem right to invoke it in code/scripts that bypassed that middleware.

It was ok. Never really failed and didn't affect the user-initiated CRUD ops though ofc the bg jobs ran when the users slept. Still wouldn't bet my life on it.

Re: Do you need separate systems when you already have Postgres?

#84

Post by ivolimmen »

Two articles on the site are already bookmarks in my app. I completely agree. Not all applications become hugely populair to actually need all those tooling. Plus since I always work in a DevOps team installation and maintenance of all those things are on our team; I don't want to learn to maintain and install all those things. If a team member suggests to add some random tool; the first thing I ask is who will be maintaining it? I'm not.

Re: Do you need separate systems when you already have Postgres?

#85

Post by ozim »

ComputerGuru wrote: post #17

I am very much in camp “minimize your dependencies and take the utmost advantage of what you already have” but I also am in camp “your database is the most important bottleneck component.” This has two implications: 1) make sure if you use Postgres for anything beyond core rdbms functionality that there is no dependency between the two, so you can rip out the additional functionality and move to a different platform…

Basically people can do it in Postgres but at least run separate instances for each purpose. Don't just throw all in single instance.

Re: Do you need separate systems when you already have Postgres?

#86

Post by CodesInChaos »

devin wrote: post #10

> This isn't about dogma. Sometimes you genuinely need specialized infrastructure. But the bar should be high: only after pushing Postgres to its limits, documenting why it was insufficient, and accepting the operational cost of the alternative. I've seen a few "Use Postgres for Everything!" posts lately. It seems to be fashionable. It reminds me of the Choose Boring Technology[1] thing from 2018 or so, but more spec…

I like keeping the number of stateful systems minimal, since those are the most annoying from a reliability perspective. In practice I use one database and object storage (S3). If that database is postgres is secondary.

Re: Do you need separate systems when you already have Postgres?

#87

Post by sarchertech »

PaulHoule wrote: post #78
sarchertech wrote: post #77

Earlier quoted context omitted.

The license you linked was too much for me because it included limitations 100 GB of data and providing audit logs for the company to inspect. But I don’t have a problem with people inserting clauses to prevent Amazon from taking over. I don’t expect free work from people forever. If I’m going to use an open source project to build a commercial product, I would only do so if I’m ok forking and maintaining the project…

The license wasn't like that when I first got involved or I wouldn't have gotten involved. I don't have any fear that Postgres will get relicensed with a worse license than it has. But I see any relicensing or license that is more restrictive than a standard license as a slippery slope that makes me think "I don't want to invest in this platform" thanks to that experience. I'm a software developer, not a lawyer. I un…

There's a couple of different issues here.

The first is that there are many what most people would call standard licenses that contain tons of restrictions. Many of them like AGPL (and GPL to a lesser extent) aren't even fully understood by lawyers.

Second, when people are talking about standard open source licenses, they are generally talking about licensees approved by the OSI. The problem is that the OSI is primarily financed by tech companies including Amazon, Microsoft, and Google. They're not ever going to add a new category of licenses or promote anything that harms those companies business interests.

As far as relicensing being a slippery slope. There's no such thing as relicensing. Once software is released under a certain license, it is always available under that license.

Anyone can choose to license future work using a different license. That is functionally equivalent to the original authors of a project ceasing work, and someone else starting work on a new fork with a different license. Nothing about Postgres using a standard license prevents this (for what it's worth Postgres is released under its own custom "PostgreSQL License" even if it is very similar to other open source licenses).

What you are actually worried about is that someone stops doing free work, which is the practical consequence of "relicensing". If that's the case you need to look into much more than the license of a project, you need to look into size of the organization, funding, size of the community, time in operation etc... I don't know if operating under a non-standard license is even in any way correlated with an organization or person relicensing or any other form of ceasing development.

Re: Do you need separate systems when you already have Postgres?

#88

Post by samuelknight »

deepsun wrote: post #45
dpc10 wrote: post #13

I love Postgres, and I agree with the general sentiment. But I read the (growing) genre of "use Postgres for everything" articles and they imply a difficulty in running other software that I just don't see. I'm thinking of Redis in particular. If you're using it as incredibly fast but not critical storage, it's trivial to set up and it ~never crashes or requires maintenance. It creates no headaches, and in exchange g…

And Redis has auto-delete rows (aka TTL). In Postgres you need a cron job to clear stale rows.

You can create an index on an expires_at column and opportunistically DELETE on each write to the cache table.

Re: Do you need separate systems when you already have Postgres?

#89

Post by otabdeveloper4 »

nilamo wrote: post #38
otabdeveloper4 wrote: post #16

Postgres sucks. It does a little bit of everything, but badly and with much manual intervention.

Wild statement, imo. There is no better open source database, so I'm very curious what you prefer and what your use cases are...

Depends on what you want.

Postgres doesn't do replication, HA or columnar/OLAP workloads, for example.

Re: Do you need separate systems when you already have Postgres?

#90

Post by deepsun »

samuelknight wrote: post #88
deepsun wrote: post #45

Earlier quoted context omitted.

And Redis has auto-delete rows (aka TTL). In Postgres you need a cron job to clear stale rows.

You can create an index on an expires_at column and opportunistically DELETE on each write to the cache table.

[dead]
Post reply on HN