Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye integers, hello UUIDv7

buildkite.com

Post reply on HN
Print view View original

241–250 of 376 posts

Re: Goodbye integers, hello UUIDv7

#241

Post by techdragon »

RhodesianHunter wrote: post #237
techdragon wrote: post #214

Earlier quoted context omitted.

This is why I’ll probably just always use a UUIDv7 primary key and a secondary UUIDv4 indexed external identifier… which is extremely close to how I tend to do things today (I’ve been using ULID and UUIDv4)

Why not use snowflake IDs?

Lookup speed. Direct mappings are faster. If something needs an external identifier that can be looked up for URL/API queries and such things…

Internal sorts and index on write with ULID/UUIDv7 which reveal potentially sensitive timestamp information, and so when that’s not appropriate a separate column can be used for…

External opaque identifiers which are UUIDv4, and if indexing speeds become an issue, can be switched to deferred indexing…

It’s a good balance and everything I’ve ever used supports this (I only needed a pretty strong representation wrapper for ULIDs since PG doesn’t validate UUID bit field structures so i can just miss-use UUID columns)

It looks like Snowflake has the same information exposure issues that using UUIDv7 or ULID for a publicly visible identifier.

Re: Goodbye integers, hello UUIDv7

#242

Post by sgarland »

imiric wrote: post #97
jonhohle wrote: post #35

This is great for internal distributed systems where having ordered keys is useful, however, it should probably be noted that these probably shouldn't be used as public identifiers (even though this will probably be the defacto standard and used publicly without thought). Having any information, specifically time information, leaking from your systems may or may not have unanticipated security or business implication…

I'm a fan of Cuid2[1] for this reason. They are compact, don't leak information, and make a good case why k-sortable IDs are unnecessary, or even harmful for performance. I'm using sequential integers and created_at/updated_at timestamps for internal use, and Cuid2 IDs externally. [1]: https://github.com/paralleldrive/cuid2

The comments on performance are utterly incorrect, modulo discussions on hotspots, but you shouldn’t be sharding randomly anyway. If you get to the point where you _need_ to shard for anything other than geolocality, doing so randomly will rapidly reveal your hotspots.

> One reason for using sequential keys is to avoid id fragmentation, which can require a large amount of disk space for databases with billions of records.

Disk is cheap but not free at higher tiers. But more importantly, record fragmentation means more pages (unless you take the time to do a full table lock and rewrite it, and who’s doing that?) which means more index bloat. I assure you, that adds up once you’re into the billions of records level.

> the ids will be generated in a sequential order, causing the tree to become unbalanced, which will lead to frequent rebalancing.

Given the width of B+trees used in DBs, I doubt they generally need to go more than one or at most two levels up. I’ll take the ability to rapidly follow the leaf nodes and have a good shot at sequential reads in cache from prefetch, thanks.

Re: Goodbye integers, hello UUIDv7

#243

Post by sgarland »

dalore wrote: post #203
noduerme wrote: post #92

Earlier quoted context omitted.

Thinking that harder-to-guess IDs will mitigate attacks is an example of security by obscurity. It's better to think of any IDs in your database as being public knowledge, because they will leak anyway. Assuming that no one can guess another ID leads to shoddy practices. I generally keep IDs sequential and build security around the basic assumption that IDs are not keys, passwords, sessions, or secrets - they're just…

Having sequential ID's is more than just a security risk, it's an information risk. Competitors can use them to estimate the size of your business, the number of customers you have, and all sorts of stuff. This was used in the war to estimate the number of German tanks based on the sequential IDs https://en.wikipedia.org/wiki/German_tank_problem So just for business intelligence you don't want to leak your IDs.

I’ve heard this argument many times, but I’ve never seen anyone actually post a reference to it happening (as in, a company finding and using this information; not the German tank problem).

To me, it reeks of solving imaginary problems while causing new ones.

Re: Goodbye integers, hello UUIDv7

#244

Post by tveita »

rockwotj wrote: post #3

I find it interesting that it’s quoted random IDs are bad for performance, because it’s actually better for distributed storage systems because you don’t hotspot on a single node. For example see: https://stackoverflow.com/a/53901549 and https://medium.com/google-cloud/cloud-spanner-choosing-the-r...

It's great for performance, up until you reach the point where a single device becomes a bottleneck, at which point it's terrible for performance.

As a sibling comment says, you ideally want to shard on some other key to get "just enough" distribution that all your machines/disks have work to do, but you are still only hitting a limited number of hot sectors on each disk that can be effectively cached. But that requires active monitoring and rebalancing of your data as it grows. Totally random keys are a safe default that will scale with any kind of data distribution and access patterns.

Re: Goodbye integers, hello UUIDv7

#245

Post by ilyt »

mort96 wrote: post #233
londons_explore wrote: post #193

Earlier quoted context omitted.

But the private data you are protecting (the user's account creation time) has the same properties as an eternal secret. Therefore there doesn't seem to be much downside in this specific case.

If you were using something like UUIDv4, you wouldn't be exposing that information at all though, neither in cleartext or ciphertext. It seems weird to say, "the user ID contains secret information so we encrypt it with an eternal fixed pre-shared key then share the ciphertext with the world", when you could've just said "the user ID contains no secret information". It feels like the right solution here is to pick be…

You could just randomize the timestamp. adding +/- month or two to the UUIDv7 won't break the advantages all that much.

Re: Goodbye integers, hello UUIDv7

#246

Post by RhodesianHunter »

afavour wrote: post #42
andersa wrote: post #27

Earlier quoted context omitted.

There are definitely many cases where it isn't an issue since you were going to tell the user the time anyway (like sent time on a message)

Can’t agree with that logic. Unless it’s specifically documented leaking timestamp data is going to get totally forgotten. So when you add (e.g.) the ability to change the sent timestamp on a message you’re going to inadvertently leak when a timestamp has been changed. Could cause embarrassment in a lot of scenarios.

This sounds like a completely arbitrarily invented argument against a technology that's perfectly useful in many scenarios.

Re: Goodbye integers, hello UUIDv7

#247

Post by Karellen »

yard2010 wrote: post #150
littlestymaar wrote: post #99

Earlier quoted context omitted.

> soviets played soccer and Cubans played baseball It's a Henri Kissinger's quote, but it's not accurate: Cubans do in fact play football. Also this quote wasn't from the 1962 Cuban missile crisis, but to another event in 1970. That being said, it is true that the US intelligence got warned by the construction of football fields (or maybe even more so by the lack of baseball grounds). https://www.cracked.com/article_…

That's Goodhart's law, if you want to know something instead of asking just say something which is wrong and someone will correct you

lol

iswydt

Re: Goodbye integers, hello UUIDv7

#248

Post by dbbk »

sgarland wrote: post #243
dalore wrote: post #203

Earlier quoted context omitted.

Having sequential ID's is more than just a security risk, it's an information risk. Competitors can use them to estimate the size of your business, the number of customers you have, and all sorts of stuff. This was used in the war to estimate the number of German tanks based on the sequential IDs https://en.wikipedia.org/wiki/German_tank_problem So just for business intelligence you don't want to leak your IDs.

I’ve heard this argument many times, but I’ve never seen anyone actually post a reference to it happening (as in, a company finding and using this information; not the German tank problem). To me, it reeks of solving imaginary problems while causing new ones.

It's perhaps embarrassing for a new startup to have a user ID of "10".

That's about the only problem I can discern.

Re: Goodbye integers, hello UUIDv7

#249

Post by RhodesianHunter »

dajonker wrote: post #115

Similar to the old situation in the article, we are using sequential 64 bit primary keys, but we use an additional random 64 bit key for external usage (instead of 128 bit). The external key is base64 encoded for use in URLs which results in an 11 byte string. This hides any information about the size of the data, the creation date of customer accounts (which would be sort of visible with UUIDv7) and prevents anyone…

What's the risk of collisions with your external ID in this scenario?

Re: Goodbye integers, hello UUIDv7

#250

Post by dbbk »

gabereiser wrote: post #54
perfmode wrote: post #21

Earlier quoted context omitted.

What are the benefits of using the Postgres uuid type (versus using TEXT or VARCHAR)?

Stored in binary format, validation, more efficient due to non-cast, faster access due to non char*, being able to split the high-low, indexing and uniqueness at the byte level.

What validation? "Postgres doesn’t care what you store in it as long as it has the correct length."
Post reply on HN