Live data from Hacker News

Choosing a Postgres primary key

supabase.com

51–60 of 163 posts

Re: Choosing a Postgres primary key

#51
post #9

While this is a good overview of the options for primary key generation, there's no silver bullet here. Most projects that are using SQL should just use the gold standard: an auto-incrementing integer for an internal primary key. And then decouple the public-facing primary key from it into a separate column, whether it be ULID, UUID, or a random-project-slug-123. Also, during debugging, it's a lot nicer to look at sh…

I hate UUIDs with passion currently - not postgres but recently spent so much extra time on relatively small table (35 mil records in few columns) and doing some queries and updating subset of it.

UUIDs there is stored in Oracle 'raw' datatype which to me is the worst combination possible, basically string stored in small binary blob, due to binary nature all needs to be converted to hex all the time for matching and readability, atrocious performance on stored procedures. Absolutely worst DB design I've seen in past 20 years, and we talk about expensive core anonymization service of top big banking package.

Re: Choosing a Postgres primary key

#52
post #6

My opinion. Always if in any way possible pick a semantic key. There is usually something defining the thing you are working on. If there isnt work on your normalisation. Main benefits to this: Avoids accidental duplication (happens so much). Avoids additional round trips to fetch the id to make a mutation. Of course if you work on something where you don’t know what it is yet (actually humans are a good example for…

To add to the other comments mentioning why this is difficult in practice:

It's the "Ship of Theseus" paradox [1]. Choosing a semantic key means mixing identity and attribute, while a synthetic key solves by assuming "constitution is not identity".

Since a digital system is a model of the world, a synthetic key allows the system to address objects in this internal model without assuming a particular interpretation of identity in the real world. E.g., it's often the case that you do need to have two "customer" entries in your system that represent the same physical "person" in the world, and this is ok because the concept of "customer" is useful and sufficient for your model, and the physical person isn't.

More often than not, people get caught on this trap in relational databases and object oriented modelling. This can be seen in books and lectures that use Customer-Order-Product relations to teach databases, or Car-Engine to teach OO.

[1] https://en.wikipedia.org/wiki/Ship_of_Theseus

Re: Choosing a Postgres primary key

#53
Funny this keeps coming up!

I wrote about my experience using ulids the other day, specifically with Postgres and some of the dis/advantages you get with it.

It's a deeper dive into ulids than this article is, and shows some real world issues that crop up:

https://blog.lawrencejones.dev/ulid/

That said, and spoiler alert: I'd probably go with bigint-sequence backed text IDs if I were choosing this over again.

Re: Choosing a Postgres primary key

#54
post #33

Disclaimer: not a dba so my terms might not be appropriate I’ve seen uuid4 which replaces the first 4 bytes with a timestamp. It was mentioned to me that this strategy allows postgres to write at the end of the index instead of arbitrarily on disk. I also presume it means it has some decent sorting. [inspiration]( https://github.com/tvondra/sequential-uuids/blob/master/sequ... )

I use ULID, 128 bits, time and great sorting https://github.com/ulid/spec

Wrote about my experience using ulids in Postgres if people are considering it: https://blog.lawrencejones.dev/ulid/

Re: Choosing a Postgres primary key

#55
post #10

Honestly that's a poor blog post. Randomly concludes "the best time-based ID seems to be xid" without saying why or comparing to others e.g. ksuid, UUIDv7 etc ("xid" is only mentioned twice in the entire blog, first in the above statement and second a link to the reference implementation). Equally unfortunate that they picked "xid" as their supposed "best" because Postgres has an internal identifier that is also call…

Collaborative databases (Wikidata, TheMovieDB, VNDB, etc.) all use serial identifiers. What is the problem with this? These websites don't want to hide how many entries they have (they tend to promote them), and it doesn't really matter if you iterate through all the numbers – the data is available through open licences anyway. I think there are many situations where you don't want to expose predictable identifiers,…

> What is the problem with this?

It causes meetings with people who think all predictable identifiers are a problem.

Re: Choosing a Postgres primary key

#56
There is an umentioned security aspect that you should be aware of for adopting timestamp-based ids: you are leaking information about time, and this information could be sensitive.

This is how I would summarize a security perspective.

    * autoincrement id: leaks information about the system as a whole. Users can attack each other. Might be suitable for an internal-only application or an application that doesn't care about leaking this information and goes to great effort to be resilient to users attacking each other.
    * timestamp + random id: leaks information about the time the individual record was created. An attacker can attempt to learn sensitive information about an individual. Suitable for a record that is already publicly shared with its time (e.g. a tweet). Might be suitable otherwise if ids are not public. That is only the record creator can view the id and you don't send out links with the ids to the user (particularly over insecure channels such as email).
    * random id: does not leak information. suitable for any use case that is okay with the performance implications (of a non-sortable fragemented index).
I am wary of how they call xid the best time-based id. It just removes all (run-time) randomness and thus performs the best. xid seems to be the same as MongoDB's oid. It is designed to be a conflict-free timestamp that can be used in a distributed system, and it is good at that. But in terms of protecting users for some use cases it could be worse than an auto-increment id because cross-user attacks are still possible (they will take many, many more attempts though) and it leaks information about time.

Re: Choosing a Postgres primary key

#57
post #43
post #33

Earlier quoted context omitted.

I use ULID, 128 bits, time and great sorting https://github.com/ulid/spec

Is there any way to have the database generate these automatically vs your application?

Yea, there are a few extensions for PG, in C and Go that give a ulid_create() function that can be used as column default, just like serial.

Re: Choosing a Postgres primary key

#59
post #40
post #30

Earlier quoted context omitted.

Not necessarily. The idea here is that the id can be exposed publically because it is random. The problem it solves is someone sees a page /accountdetails?id=123 and can easily look for /accountdetails?id=124 and assume it is likely to be valid. If you use a random id, you cannot quickly know what other ids exist which makes looking for unauthorized access to objectids much harder.

If there needs to be an auth check for other IDs, then it shouldn't matter whether the IDs are random or not. Depending on how highly aesthetic URLs are valued, it's not unlikely that after being in business for a while, the density of your keyspace will mean that even random IDs are found. A better rationale for disconnecting public and private IDs is to make certain types of database migration a little bit easier.…

https://en.m.wikipedia.org/wiki/German_tank_problem

Preventing competitors from estimating the size of your business (or of your customer's businesses, if generating sequential IDs on their behalf) is one big reason for having unguessable public IDs.

Re: Choosing a Postgres primary key

#60
post #6

My opinion. Always if in any way possible pick a semantic key. There is usually something defining the thing you are working on. If there isnt work on your normalisation. Main benefits to this: Avoids accidental duplication (happens so much). Avoids additional round trips to fetch the id to make a mutation. Of course if you work on something where you don’t know what it is yet (actually humans are a good example for…

Semantic key keys have downsides, and it is also important to note that there is not always a single unique semantic key.

Relation theory has the concept of superkeys, which are a set of columns that uniquely determine a row. But these are not useful, since for example, the set all all columns should uniquely determine a row. What is useful is "candidate keys", which are minimal superkeys, with any columsn not necessary to be unique removed.

There are two ways to determine candidate keys. One is empirically by analyzing the data. If you do it that way, then the "candidate" naming is appropriate, since it is possible that some sets of columns are unique simply by chance, not by fundamental nature, and unique by chance is not what we want.

Alternatively you can use domain knowledge and logic to determine candidate keys. Candidate keys determined by logic are true keys, since no duplicates should ever occur unless the requirements or fundamental nature of the data changes. This means that ideally, all such keys should have a unique constraint placed on them (although the implicit unique constraint from marking as a primary key will works for one of these keys). Adding unique constraints for all logically determined candidate keys is the ideal way to avoid accidental duplication.

Within the database and within the application, you ideally want to only use small keys that are unlikely to change, and are unlikely to ever become non-unique. Keys that change tend to cause headaches with updates if referenced elsewhere, and you can have undesirable race condition issues with application logic on changing keys.

Similarly, for keys likely to become non-unique from changing requirements, using them within the database means a much bigger refactor later if they become no longer unique. But if you never use those value to reference within the database, then simply dropping the unique constraint is easy. Impact on application code may vary, from potentially no change needed at all, to much more significant changes, depending on the data in question and how the application uses it.

Large keys that don't change, and are extremely unlikely to ever become non-unique are conceptually fine, but have the practical problem of being large, and thus undesirable to reference from all over the database from a file size perspective. This is especially true of multi-column keys which also tend to be inconvenient from a query writing perspective.

Another important issue is that many identifiers that are supposed to be universally unique, like UPCs, ISBNs etc, are not actually always unique. These things do end up getting occasionally reused, usually accidentally. If you are using that everywhere as your primary key, and eventually come across such a scenario, it is a real nightmare to refactor everything to use a different key in order to be able to handle this. While if you are using some surrogate key almost everywhere, it becomes a lot more feasible to handle this with things like having "lookup by UPC" screens show a list of options when you stumble upon one that happens to have a duplicate.

Post reply on HN