Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye integers, hello UUIDv7

buildkite.com

Post reply on HN
Print view View original

171–180 of 376 posts

Re: Goodbye integers, hello UUIDv7

#171

Post by gregw2 »

jillesvangurp wrote: post #108
rockwotj wrote: post #3

I find it interesting that it’s quoted random IDs are bad for performance, because it’s actually better for distributed storage systems because you don’t hotspot on a single node. For example see: https://stackoverflow.com/a/53901549 and https://medium.com/google-cloud/cloud-spanner-choosing-the-r...

Depends how you query it. In a lot of systems, recently added data is also the most queried data or data typically gets pulled out sorted by time. Having that data on disk in more or less the order it is going to be queried makes sorting it a bit easier. Even in a sharded system, each of the shards would have less work to do for sorting. Of course a lot of these systems would have an append only write model which wou…

An explicit shard id can ensure all related data across all tables can be on the same shard. Helpful for SQL JOIN operations.

Picking the N least significant bits only a single table has good distribution and sort qualities, no cross-table properties.

Re: Goodbye integers, hello UUIDv7

#172

Post by robertlagrant »

nindalf wrote: post #163
oconnore wrote: post #55

Earlier quoted context omitted.

Given that a UUID identifier fits in a single cipher block, and the whole point is that these are unique by construction (no IV needed so long as that holds true), it seems like a single round of ECB-mode AES-128 would enable quickly converting between internal/external identifiers. 128 bits -> 128 bits

One key for all tokens or one key per token? If it’s the latter a simple XOR would do because it would be the equivalent of a one time pad.

I don't think it can be a key per token, or it will scale appallingly.

Re: Goodbye integers, hello UUIDv7

#173

Post by amelius »

oconnore wrote: post #143
dragontamer wrote: post #139

Earlier quoted context omitted.

Why not just use the AES-128 result as the UUID then? What's the benefit of the internal structure at all? If AES-128 is an acceptable external UUID (and likely an acceptable internal one), then you might as well just stick with a faster RNG.

That would be the same as using a random identifier (UUIDv4, for example) with the associated indexing issues when stored in a database. The whole point here would be that you can expose something opaque externally but benefit from well behaved index keys internally.

Storage is cheap, you might as well store the extra integer.

Re: Goodbye integers, hello UUIDv7

#174

Post by rvnx »

jonhohle wrote: post #35

This is great for internal distributed systems where having ordered keys is useful, however, it should probably be noted that these probably shouldn't be used as public identifiers (even though this will probably be the defacto standard and used publicly without thought). Having any information, specifically time information, leaking from your systems may or may not have unanticipated security or business implication…

It’s as convenient as manipulating IPv6 addresses.

Re: Goodbye integers, hello UUIDv7

#175

Post by Nevermark »

nindalf wrote: post #163
oconnore wrote: post #55

Earlier quoted context omitted.

Given that a UUID identifier fits in a single cipher block, and the whole point is that these are unique by construction (no IV needed so long as that holds true), it seems like a single round of ECB-mode AES-128 would enable quickly converting between internal/external identifiers. 128 bits -> 128 bits

One key for all tokens or one key per token? If it’s the latter a simple XOR would do because it would be the equivalent of a one time pad.

One key per token would require a table matching internal tokens to their key for forward conversion, and another table matching external keys to their key for reversing.

Might as well just use randomly generated external keys and have one table if you were doing that.

So, one key per all tokens.

Re: Goodbye integers, hello UUIDv7

#176

Post by masklinn »

amelius wrote: post #173
oconnore wrote: post #143

Earlier quoted context omitted.

That would be the same as using a random identifier (UUIDv4, for example) with the associated indexing issues when stored in a database. The whole point here would be that you can expose something opaque externally but benefit from well behaved index keys internally.

Storage is cheap, you might as well store the extra integer.

Storage is cheap, updating indexes is not.

Re: Goodbye integers, hello UUIDv7

#177

Post by andix »

rockwotj wrote: post #3

I find it interesting that it’s quoted random IDs are bad for performance, because it’s actually better for distributed storage systems because you don’t hotspot on a single node. For example see: https://stackoverflow.com/a/53901549 and https://medium.com/google-cloud/cloud-spanner-choosing-the-r...

There are many other options that usually scale better than random distribution. For example distribution by user or tenant id.

Re: Goodbye integers, hello UUIDv7

#178

Post by masklinn »

oconnore wrote: post #146
MattPalmer1086 wrote: post #137

Earlier quoted context omitted.

This seems overly complex, and you need some kind of key too. Why not just hash it with pretty much any hash function?

A hash is not reversible, so you’d need a database index to recover the original efficient-to-index identifier, which misses the whole point ;) If you didn’t care about index clustering then just use UUIDv4

Although you could use a hash index to avoid the more deleterious effects of insertion, as long as you don’t need a unique constraint anyway…

Re: Goodbye integers, hello UUIDv7

#179

Post by Foobar8568 »

erik_seaberg wrote: post #114
Foobar8568 wrote: post #95

Earlier quoted context omitted.

Clock skews in a single node database? Actual question.

Usually Hive or Dremel, with rows ingested from frontend instance logs. I get that it would actually work on a smaller system with a single server assigning times (or a quorum, if leader election ensures a monotonic clock).

Thank you, I was just wondering if it was something I missed out on simpler systems and not distributed ones.

Re: Goodbye integers, hello UUIDv7

#180

Post by andix »

oconnore wrote: post #55
jonhohle wrote: post #35

This is great for internal distributed systems where having ordered keys is useful, however, it should probably be noted that these probably shouldn't be used as public identifiers (even though this will probably be the defacto standard and used publicly without thought). Having any information, specifically time information, leaking from your systems may or may not have unanticipated security or business implication…

Given that a UUID identifier fits in a single cipher block, and the whole point is that these are unique by construction (no IV needed so long as that holds true), it seems like a single round of ECB-mode AES-128 would enable quickly converting between internal/external identifiers. 128 bits -> 128 bits

I like the idea, but I think it's not possible to rotate the key with that approach, without introducing a breaking change. Eternal secrets are usually a very bad idea, because at some point they are going to be leaked.
Post reply on HN