Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye integers, hello UUIDv7

buildkite.com

Post reply on HN
Print view View original

291–300 of 376 posts

Re: Goodbye integers, hello UUIDv7

#291

Post by atonse »

samatman wrote: post #285

Relying on timestamps to be sortable, when clock skew and ntd guarantee that they won't always be, strikes me as poor design. If you need to sort by insert order, use an autoincrementing integer, if you need uniqueness, UUIDv4 is fine, if you need both use both. Use timestamps when you need to record the time, just don't commit the sin of presuming that clock time will never run backwards, I assure you, it does.

That's still fine. Because even if there's skew and other such things going on, it's more likely to take advantage of cache locality since the page in the index that key would be stored in, is much more likely to still be in memory.

Re: Goodbye integers, hello UUIDv7

#292

Post by HALtheWise »

kiitos wrote: post #278
contravariant wrote: post #238

Earlier quoted context omitted.

If you need more than second precision then millisecond doesn't get you much further. The fact that the epoch ends in 120 years is a bit more worrying, but is also just about non-critical enough that it will be ignored for at least the next century. Also, to all future historians of 2150, sorry about the mess, but yes we knew this was going to happen. Whatever it was.

> If you need more than second precision then millisecond doesn't get you much further. It gets you precisely 100x further.

*1000x

Re: Goodbye integers, hello UUIDv7

#293

Post by TRiG_Ireland »

vidarh wrote: post #224
dalore wrote: post #203

Earlier quoted context omitted.

Having sequential ID's is more than just a security risk, it's an information risk. Competitors can use them to estimate the size of your business, the number of customers you have, and all sorts of stuff. This was used in the war to estimate the number of German tanks based on the sequential IDs https://en.wikipedia.org/wiki/German_tank_problem So just for business intelligence you don't want to leak your IDs.

I agree just knowing the number is bad, but it also makes it easier to discover far worse problems as well. My second job was for a company that provided internet enabled phone conferencing solutions (this was years before VoIP became widespread). The customer ids were sequential. Couple that with an outright idiotic security flaw (the login process set the customer ID in a cookie and the app trusted it on ever subse…

Huh. That's exactly the same security flaw as Moonpig had. Tom Scott made a video about it.

Re: Goodbye integers, hello UUIDv7

#294

Post by KMag »

lifthrasiir wrote: post #164
notpushkin wrote: post #91

Earlier quoted context omitted.

Is there a way to encrypt 122 bits -> 122 bits? If so – do that and set version to 4. Alternatively, just say it's a random string ID and not an UUID.

For your information, yes, you can [1]. For example if you have a good enough 128-bit block cipher (e.g. AES-128-ECB), start with a block of 128 bits where specific 6 bits are filled out and others are filled with the plain text. Repeatedly encrypt the block until those specific 6 bits are reached again (and do the same thing in reverse for decryption). This is possible because a good block cipher is also a good pseu…

Cycle-walking FPE has a (nearly) unbounded upper-bound on latency.

Breaking the 122 bits into two 61-bit halves and using AES as the round function for a Feistel cipher gives you a constant 3-encryption latency instead of the expected 64-encryption average latency of the cycle-walking format preserving encryption.

Alternatively, use AES in VIL mode ( https://cseweb.ucsd.edu/~mihir/papers/lpe.pdf ).

Re: Goodbye integers, hello UUIDv7

#295

Post by contravariant »

mort96 wrote: post #240
contravariant wrote: post #235

Earlier quoted context omitted.

And if you consider knowledge of the id sufficient for access. Which, despite the fact that it really shouldn't be, still seems to occur every so often. Even in situations where the ids are very much not random. Honestly if I have to read one more article about a 'hacker' who 'leaked' some secret government piece ahead of time because they thought to increment the date in the url of some yearly report, I'm going to l…

I don't understand. What part of this requires that one considers knowledge of the ID sufficient for access? And what kind of access are you talking about? The performance benefits of index friendly user IDs seem like they would apply even if all user info is secret and requires a token to access... The application still has to look up the user by ID after all? If I imagine a basic authenticated "get information abou…

> Yet a user of the API still needs both the user ID and a token to access anything.

Ideally yeah.

In practice, it varies...

Re: Goodbye integers, hello UUIDv7

#296

Post by KMag »

akoboldfrying wrote: post #229
lifthrasiir wrote: post #164

Earlier quoted context omitted.

For your information, yes, you can [1]. For example if you have a good enough 128-bit block cipher (e.g. AES-128-ECB), start with a block of 128 bits where specific 6 bits are filled out and others are filled with the plain text. Repeatedly encrypt the block until those specific 6 bits are reached again (and do the same thing in reverse for decryption). This is possible because a good block cipher is also a good pseu…

What a cute observation! If there's any way for the client to influence the input, it may be prone to DoS attacks: By my calculations, with a million random attempts, you would expect to find a cycle of length at least 435, which is over 13x the average. (Mind you, multiplying the number of attempts by 10 only adds about 72.5 to the expected cycle length, and probably no one has the patience to try more than 100 bill…

The properties of the permutation are dependent upon the encryption key, so a client being able to select malicious inputs to get long cycles implies either that the client knows the AES key, or that the client has broken AES.

In any case, as I mentioned in a sibling comment, with 3 AES encryptions one can construct a 122-bit balanced Feistel cipher with a constant amount of work.

Re: Goodbye integers, hello UUIDv7

#297

Post by johncolanduoni »

kijin wrote: post #52
stepanhruda wrote: post #23

Earlier quoted context omitted.

Yes but if that machine with sequential data receives 100x the traffic of other machines, it can be worse than splitting this traffic evenly across all available machines.

If your database simply shards keys sequentially, it's going to get hotspots in a lot of use cases, like plain old integer keys and timestamps, not just UUIDv7. In that case it would be fair to say that your database is doing it wrong. Fortunately, there's no rule that says you should shard your keys using the sequential part up front. One of the rules for generating randomness from environmental sources is to throw…

What distributed databases shard on the low bits? How do they do something like a range query?

The closest I’ve ever heard of is sharding based on a hash (e.g. CockroachDB can do this on request[1]) but most distributed databases with strong consistency (Spanner descendants in particular) default to “doing it wrong”.

[1]: https://www.cockroachlabs.com/docs/stable/hash-sharded-index...

Re: Goodbye integers, hello UUIDv7

#298

Post by 0xEFF »

jonhohle wrote: post #35

This is great for internal distributed systems where having ordered keys is useful, however, it should probably be noted that these probably shouldn't be used as public identifiers (even though this will probably be the defacto standard and used publicly without thought). Having any information, specifically time information, leaking from your systems may or may not have unanticipated security or business implication…

Every access and id token issued by oidc already has an issued at (iat) and expiration (exp) fields.

Re: Goodbye integers, hello UUIDv7

#299

Post by Pxtl »

Frustrating, I looked up MS/C#'s implementation and they don't get stored in a proper semisequential fashion in MS SQL Server because MS stores UUIDs in an odd binary format.

Re: Goodbye integers, hello UUIDv7

#300

Post by samatman »

jmmv wrote: post #290
samatman wrote: post #285

Relying on timestamps to be sortable, when clock skew and ntd guarantee that they won't always be, strikes me as poor design. If you need to sort by insert order, use an autoincrementing integer, if you need uniqueness, UUIDv4 is fine, if you need both use both. Use timestamps when you need to record the time, just don't commit the sin of presuming that clock time will never run backwards, I assure you, it does.

The problem they describe is not about sorting: it’s about data locality. And for the latter, clock skew should not be a problem. Even with significant clock skew, data will end up clustered anyway, much better than with a random spread.

You get index locality with an autoincrement also, and it will actually reflect insert order. My point is that a timestamp won't do that, and worse, it will appear to most of the time. The failure can be fairly spectacular, a Unix clock can be set to any time at all, and it's good when the resulting bugs are limited to time. Having an actual insert order can be a real boon to figuring out what happened.

I hold to the principle that relational data should be normal, and combining uniqueness with a timestamp doesn't do that. To do any of the calculations we use timestamps for, you have to strip off the entropy, this complicates pushing it down to the database level, where the libraries don't expect such conflation.

You're going to have a bad time writing something like a join across tables with a restricted range of time if your time is embedded in UUIDv7.

I maintain this is good advice: if you need index locality and insert order, use an autoincrement. If you need to record and work with time, use a timestamp. If you need global uniqueness, you can use any of the UUIDs, but v4 is the one that doesn't conflate uniqueness with unrelated properties, and should be preferred.

If you think your need data locality but not insert order, think long and hard about what you're doing, because odds are you're wrong. If it turns out you're right, and the OP might be in that situation, sure, go ahead and use UUIDv7.

Just, please, for the sake of your future self and everyone you work with, don't use a timestamp for insert order. Ever.

Post reply on HN