Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye integers, hello UUIDv7

buildkite.com

Post reply on HN
Print view View original

231–240 of 376 posts

Re: Goodbye integers, hello UUIDv7

#231

Post by nhoughto »

phkahler wrote: post #227
nhoughto wrote: post #221

Earlier quoted context omitted.

Yep have used an approach just like that, worked quite well if you have a strong pattern to easily translate from one to the other. Gives you an id with the right properties for internal use, efficient indexing etc, and in its encrypted form gives you the properties you want from an external identifier being unpredictable etc, all from one source id. It is true that now your encryption key is now very long lived and…

>> It is true that now your encryption key is now very long lived and effectively part of your public interface No need to encrypt, just store the external key in a table. Not that you're likely to change algorithms.

True you could rotate by persisting the old value and complicate your lookup/join process, not my idea of an acceptable solution but yep totally possible and worth it for some set of tradeoffs.

Re: Goodbye integers, hello UUIDv7

#232

Post by XCSme »

jhealy wrote: post #204
erik_seaberg wrote: post #43

> first component (prefix) of the identifier is a sortable timestamp > values generated are practically sequential These statements aren’t strict enough to be relied on. Maybe you have engineered the hell out of your distributed clock scheme, and your IDs actually are completely monotonic, which is great. But you probably haven’t done that, which means conflicts will surely happen and you must handle them gracefully.

I'm not the author but I work at the same company. None of our systems require perfect ordering of IDs generated across our distributed system. Most of the system was built with random UUIDv4 identifiers so no code assumes the ID ordering is significant. However, in much of our system recent data is frequently accessed while old data is rarely accessed. In that world, just having the IDs *approximately* clustered in…

Just curious, if there is no ordering of the data (no sequential ID), how is it even "ordered" on the file system? Is it random? Or, even without a sequential ID can you know which data was written earlier and which later?

Re: Goodbye integers, hello UUIDv7

#233

Post by mort96 »

londons_explore wrote: post #193
andix wrote: post #180

Earlier quoted context omitted.

I like the idea, but I think it's not possible to rotate the key with that approach, without introducing a breaking change. Eternal secrets are usually a very bad idea, because at some point they are going to be leaked.

But the private data you are protecting (the user's account creation time) has the same properties as an eternal secret. Therefore there doesn't seem to be much downside in this specific case.

If you were using something like UUIDv4, you wouldn't be exposing that information at all though, neither in cleartext or ciphertext. It seems weird to say, "the user ID contains secret information so we encrypt it with an eternal fixed pre-shared key then share the ciphertext with the world", when you could've just said "the user ID contains no secret information".

It feels like the right solution here is to pick between: use UUIDv7 and treat the account creation time as public information, or use an identifier scheme which doesn't contain the account creation time.

Re: Goodbye integers, hello UUIDv7

#234

Post by Waterluvian »

dajonker wrote: post #115

Similar to the old situation in the article, we are using sequential 64 bit primary keys, but we use an additional random 64 bit key for external usage (instead of 128 bit). The external key is base64 encoded for use in URLs which results in an 11 byte string. This hides any information about the size of the data, the creation date of customer accounts (which would be sort of visible with UUIDv7) and prevents anyone…

It sounds like you basically just made your own 64 bit UUID. If you’re exposing this ID for manual use by a human (like URLs) then that sounds pretty helpful to be shorter!

Re: Goodbye integers, hello UUIDv7

#235

Post by contravariant »

andersa wrote: post #18

> We use sequential primary keys for efficient indexing, and UUID secondary keys for external use. The upcoming UUIDv7 standard offers the best of both worlds Unless you consider users being able to extract the generation time from the id to be an issue, of course.

And if you consider knowledge of the id sufficient for access.

Which, despite the fact that it really shouldn't be, still seems to occur every so often. Even in situations where the ids are very much not random.

Honestly if I have to read one more article about a 'hacker' who 'leaked' some secret government piece ahead of time because they thought to increment the date in the url of some yearly report, I'm going to lose my mind.

Re: Goodbye integers, hello UUIDv7

#236

Post by sgarland »

londons_explore wrote: post #202
pdpi wrote: post #181

Earlier quoted context omitted.

> What's the benefit of the internal structure at all? Purely random identifiers are the bane of DB indices. The internal structure is sequential-ish and therefore indexes well.

Purely random identifiers are the recommended primary key for plenty of databases - eg. spanner. Random identifiers spread work evenly between shards of something sharded by keyspace. They also don't get 'hotspotting' on more recent records - recent records frequently get more than their fair share of updates and changes, and database query planners have no knowledge of that.

Spanner is also bespoke, and was probably designed with that in mind.

Anything with a clustering index - MySQL (with InnoDB), SQL Server - will absolutely hate the page splits from a random PK.

Postgres also doesn’t like it that much, for a variety of reasons. Tuples are in a heap, but the PK is still a B+tree (ish), so it still suffers from splits. They also use half the default page size as MySQL, AND their MVCC implementation doesn’t lend itself to easy updates without rewriting the entire page.

Go run benchmarks with some decent scale (tens of millions of rows or more) on any DB you’d like between UUIDv4 and UUIDv7 as the PK, and see how it goes.

Re: Goodbye integers, hello UUIDv7

#237

Post by RhodesianHunter »

techdragon wrote: post #214
masklinn wrote: post #176

Earlier quoted context omitted.

Storage is cheap, updating indexes is not.

This is why I’ll probably just always use a UUIDv7 primary key and a secondary UUIDv4 indexed external identifier… which is extremely close to how I tend to do things today (I’ve been using ULID and UUIDv4)

Why not use snowflake IDs?

Re: Goodbye integers, hello UUIDv7

#238

Post by contravariant »

kiitos wrote: post #66
Lazare wrote: post #41

UUIDv7 is a nice idea, and should probably be what people use by default instead of UUIDv4 for internal facing uses. For the curious: * UUIDv4 are 128 bits long, 122 bits of which are random, with 6 bits used for the version. Traditionally displayed as 32 hex characters with 4 dashes, so 36 alphanumeric characters, and compatible with anything that expects a UUID. * UUIDv7 are 128 bits long, 48 bits encode a unix tim…

Second precision is too coarse for many (most?) use cases.

If you need more than second precision then millisecond doesn't get you much further. The fact that the epoch ends in 120 years is a bit more worrying, but is also just about non-critical enough that it will be ignored for at least the next century.

Also, to all future historians of 2150, sorry about the mess, but yes we knew this was going to happen. Whatever it was.

Re: Goodbye integers, hello UUIDv7

#239

Post by tzahifadida »

Not specifically the topic, but I looked for a library for golang and it is not that common, there is a library in https://www.postgresql.org/docs/current/uuid-ossp.html Java has something, but again not really clear how tested. So using this is a bit iffy...

Re: Goodbye integers, hello UUIDv7

#240

Post by mort96 »

contravariant wrote: post #235
andersa wrote: post #18

> We use sequential primary keys for efficient indexing, and UUID secondary keys for external use. The upcoming UUIDv7 standard offers the best of both worlds Unless you consider users being able to extract the generation time from the id to be an issue, of course.

And if you consider knowledge of the id sufficient for access. Which, despite the fact that it really shouldn't be, still seems to occur every so often. Even in situations where the ids are very much not random. Honestly if I have to read one more article about a 'hacker' who 'leaked' some secret government piece ahead of time because they thought to increment the date in the url of some yearly report, I'm going to l…

I don't understand. What part of this requires that one considers knowledge of the ID sufficient for access? And what kind of access are you talking about?

The performance benefits of index friendly user IDs seem like they would apply even if all user info is secret and requires a token to access... The application still has to look up the user by ID after all?

If I imagine a basic authenticated "get information about me" style endpoint, that would take a user ID and an authentication token. Checking if the token is valid is faster if the user ID is index friendly. Getting the requested information is faster if the user ID is index friendly. Yet a user of the API still needs both the user ID and a token to access anything.

Post reply on HN