Live data from Hacker News

Why does everyone run ancient Postgres versions?

neon.tech

371–380 of 452 posts

Re: Why does everyone run ancient Postgres versions?

#371

Earlier quoted context omitted.

To be honest, indexes aren't designed for that. They're meant for fast lookup of short identifiers. Things like people's names and product ID's. Not long URL's. It's not performant. If you need to keep a million long URL's in a defined sort order, my first recommendation would be, don't -- see if there's another way to achieve your end result. But if you absolutely have to, then create a new integer column to be your…

> indexes aren't designed for that. They're meant for fast lookup of short identifiers. Things like people's names and product ID's. Not long URL's. It's not performant. This is objectively false. If this was true, indexes wouldn't serve range queries. You couldn't index on dates. You couldn't sort numbers. > But if you absolutely have to, then create a new integer column to be your sort key, and use a little bit of…

Sorry, I was considering short things like dates and numbers as identifiers. I realize that's not quite right -- what I should have said was that indexes are designed for short things period (short identifiers being one of those things). Thanks.

> This fails when you need to insert new values into the table.

Yes, that's part of the extra code you need to keep the values accurately sorted. There are a lot of different particular code solutions that might work -- whether allowing for collisions and re-ordering every night with a cron job, or putting large gaps between numbers, or using floats.

But my main point stands, which is that standard relational databases are not designed to be able to maintain a sorted index of long URL's out of the box. Indexes aren't meant for that and they won't work, and this is by design. You're going to have to roll your own code for that.

Fortunately I've never come across a case in the wild where maintaining a globally sorted list of long items was required (though I'm not saying they never exist). E.g. if you're building a spider that needs to match against URL's, you'd index a short hash of the URL as a non-unique index. Or if you wanted to display sorted URL's for a site, you'd index by domain name only, and then sort the remainder of the URL at query time.

Re: Why does everyone run ancient Postgres versions?

#372
post #365

Earlier quoted context omitted.

That's a good list [1]. A handful of these are already doable in modern MySQL and/or MariaDB though. JSON can often be used in place of arrays, and JSON_TABLE in both MySQL and MariaDB converts JSON into tabular data. MySQL supports multi-valued indexes over JSON, where each row can have multiple index entries (or no entries, e.g. partial index). MariaDB has built-in convenience types for ipv4, ipv6, and uuid. Or in…

> JSON can often be used in place of arrays This is like storing UUIDs as text. You lose type information and validation. It's like storing your array as a comma-delimited string. It can work in a pinch, but it takes up more storage space and is far more error prone. > convenience types for ipv4, ipv6, and uuid. That's nice to see. A shame you have to decide ahead of time whether you're storing v6 or v4, and I don't…

Regarding using JSON for arrays, MySQL and MariaDB both support validation using JSON Schema. For example, you can enforce that a JSON column only stores an array of numbers by calling JSON_SCHEMA_VALID in a CHECK constraint.

Granted, using validated JSON is more hoops than having an array type directly. But in a pinch it's totally doable.

MySQL also stores JSON values using a binary representation, it's not a comma-separated string.

Alternatively, in some cases it may also be fine to pack an array of multi-byte ints into a VARBINARY. Or for an array of floats, MySQL 9 now has a VECTOR type.

Regarding ipv6 addresses: MariaDB's inet6 type can also store ipv4 values as well, although it can be inefficient in terms of storage. (inet6 values take up a fixed 16 bytes, regardless of whether the value is an ipv4 or ipv6 address.)

As for using RETURNING inside a writable CTE in MariaDB: not sure, I'd assume probably not. I must admit I'm not familiar with the multi-table pipeline write pattern that you're describing.

Re: Why does everyone run ancient Postgres versions?

#373
post #17

In Oracle, ALTER TABLE MOVE in 8i was a godsend, finally enabling a table reorganization without export/import. My timid management forbade an upgrade from Oracle 7.3.4 until 2013. It was agony to remain on that museum piece for as long as we did. I am upgrade-minded, but my management is not. I always lose. I am retiring in two years. I will not miss their problems, not at all. Edit: Oracle 10g was the last release…

Impressive! What platform were you running that on?

Re: Why does everyone run ancient Postgres versions?

#374
I'm still on 9.x in some systems and it's running great. I find this kind of sentiment a bit weird anyway: PostgeSQL 17 has been out for a couple of weeks, I'm certainly not in a rush to upgrade anything unless I need to. Never touch a running system is as valid as ever and on top of that I'm not a full-time DBA itching to upgrade as soon as possible. With containerization it's also more common to have multiple and right out many DB instances, I won't be going through all of them until someone requests it. Security updates is a completely different matter, but major versions? Don't get the rush, the developers will come around sooner or later.

Re: Why does everyone run ancient Postgres versions?

#375

Earlier quoted context omitted.

> It couldn't be easier If you have a trivial database, yes. If you have a large, complicated database that takes hours/days to restore and has an HA SLA, very hell no.

That's not going to be easy with any database.

MySQL, for all its faults, actually handles this quite well. Generally speaking, assuming there aren't any major backwards-incompatibilities (which there haven't often been in my experience), you can start up the old server in place of the new server and you're good to go. Some table-rebuilding can be done in-place if you need to do things like recreate indices, but _generally_ you're good to go from there.

For more complex deployments, cases of backwards-incompatibilities in the data structure, or where I had reason to rebuild many large tables, you set up a replication secondary, do pre-upgrade migrations (e.g. in the case of deprecated functionality), upgrade the secondary, do post-upgrade migrations (e.g. in the case of new functionality), and wait for replication to catch up 100%. Then, after sending traffic to it for a bit to ensure it works correctly, you swap the primary and secondary and do it again.

Re: Why does everyone run ancient Postgres versions?

#376

Earlier quoted context omitted.

> A distro takes care of all of those generally and makes sure the dependencies have the same care taken. So does using the vendor repos: * https://apt.postgresql.org/ / https://yum.postgresql.org * https://dev.mysql.com/downloads/repo/ This way when you upgrade your OS you don't have to worry about suddenly getting a new version (which, in the case of MySQL, may take a long time to convert its database files to the…

How does that work for python/perl/ruby libs etc?

It's very rare that the client protocol changes dramatically and in an incompatible way, so typically nothing changes and no one notices.

I've seen a few rare cases where an update to the client library is required for improvements to the protocol, e.g. when MySQL switched to the new password/authentication format, but that was also backwards-compatible: the old format was used until an account's password was changed, password changes could be told to use the legacy format instead, and you could set the legacy format by default. The only issue that occurred (and it did occur) was old client libraries + new server password hashes, but, again, in this case it didn't affect existing installations/users/passwords.

Re: Why does everyone run ancient Postgres versions?

#377

Because the actual process of upgrading Postgres is terrible. I say this as someone who absolutely loves using it, but the actual process of upgrading Postgres is something that takes significant downtime, is error-prone and you're often better off just dumping all the databases and reimporting them in a new folder. (A good idea in general since it'll vacuum and compact indexes as well if you do it, combining a few m…

I fully agree. That's why I use this: https://github.com/pgautoupgrade/docker-pgautoupgrade But obviously this is not suitable for all use cases. I don't know why pg doesn't invest in this sector.

Thanks for the link, wasn't aware of this solution.

Re: Why does everyone run ancient Postgres versions?

#378
post #8

Because upgrading is a lot of work, and is higher risk than upgrading other software.

Seems like a massive design fail if they can't maintain backwards compatability and provide a safe, low friction upgrade process.

Related, but Sandstorm is an app-hosting platform/solution that's very different in design than other solutions like Docker, and one of the reasons is that it's actually designed from the ground up for easy and effective use on homelab-style setups, because among other things (not limited to this, and 100% from memory):

1. Sandstorm apps need to always come back cleanly from a raw SIGKILL at any moment.

2. Sandstorm apps must be able to always upgrade ANY previous version of stored data, with no intervention, automatically, when newer versions are deployed.

These are tough technical constraints and have huge implications on the whole design. For example, the client/protocol layer needs to possibly be multi-version aware for clean rollouts, client connection failover needs to be built in, etc. But the reality is if you do not have these two constraints, your software will never really work in a turnkey forget-about-it-way way where someone just runs an instance and then actually forgets about it, and it can be safe and secure. This kind of stuff is imperative to every actual computer user who isn't a programming nerd.

This is why slapping Docker on a lot of existing pieces of software like databases doesn't really work in the grand scheme. Yes, it gets you started faster, which people value to a high degree. But it often lacks or has the same problems as other solutions on day 2, day 3, ... etc. So, you need to solve all the same problems anyway just in a different form (or externalize them onto some other solution.)

Realistically, the solutions have to be designed in from the ground up. That's very difficult to do but necessary.

Someone else mentioned in here that SQL Server always 100% works when upgraded in place from an old version. That's also my (limited, homelab-ish) experience and a good reason to like it.

Re: Why does everyone run ancient Postgres versions?

#379
post #185

Earlier quoted context omitted.

So the real question is, why is the upgrade process so incompetently designed, and why has no one fixed this?

My opinion is Postgres was designed by software developers for software developers. The split on “which relational database to use” in my career has almost always been perfectly split between SWE vehemently demanding pgsql for the feature set, and the sysadmins having to support maintenance and production availability preferring MySQL. One of the few things I’ve enjoyed with the move into devops and companies forcing…

MySQL feels more "modern" (in the context of the early 2000s when I started using it), in that it seemed to know it existed in an environment with other tools; for example, authentication was handled internally and was unrelated to the system user (other than the clients usually using your system username as the default database username if you didn't specify one).

Compare that with Postgres, which seemed very "old school", going so far as to assume it was the only thing a given server was doing. Connecting to postgres authenticated as your own user; creating a user was done with the `createuser` command (or similar, I don't remember what it was actually called), and not some namespaced `pg_createuser` command that would make it clear what it did.

I also remember setting up MySQL replication with almost no effort whatsoever, and then in the same year trying to set up Postgres replication - which it didn't have. I was told by other postgres admins to "just set up a script to rsync the database over and over to the other server; then if your first server dies just start the second server up and it'll recover". This seemed like a wildly cavalier attitude towards uptime and reliability, not to mention generating a ridiculous amount of I/O and network traffic for minimal benefit.

Re: Why does everyone run ancient Postgres versions?

#380
post #378

Earlier quoted context omitted.

Seems like a massive design fail if they can't maintain backwards compatability and provide a safe, low friction upgrade process.

Related, but Sandstorm is an app-hosting platform/solution that's very different in design than other solutions like Docker, and one of the reasons is that it's actually designed from the ground up for easy and effective use on homelab-style setups, because among other things (not limited to this, and 100% from memory): 1. Sandstorm apps need to always come back cleanly from a raw SIGKILL at any moment. 2. Sandstorm…

Huh, I never really thought of that as a unique property of Sandstorm, but you're absolutely right, this is essential to any software intended to be operated by a non-technical end user, and Sandstorm is aggressive about it (e.g. apps are always shut down by abrupt SIGKILL and whenever someone requested that we add some sort of clean shutdown notification, I refused).

Ironically, Sandstorm itself is still stuck running on Mongo version 2.6 (a decade old!) because Mongo can't necessarily be updated in an unattended way. Of course, we could never ask every Sandstorm user to do some manual process to upgrade it.

Meanwhile, newer versions of the Mongo Node.js client library don't support such old Mongo, which means Sandstorm cannot actually update that package nor any of the packages that depend on it anymore. And this is why Sandstorm is now stuck in time and no longer receiving updates. :(

It was clearly a big mistake for Sandstorm to use Mongo. We chose it because we built the UI on Meteor, which was really only designed to work wing Mongo. In retrospect I wish we'd used SQLite, whose backwards-compatibility guarantee has now reached twenty years.

Post reply on HN