Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye integers, hello UUIDv7

buildkite.com

Post reply on HN
Print view View original

101–110 of 376 posts

Re: Goodbye integers, hello UUIDv7

#101

Post by berkes »

noduerme wrote: post #92
jonhohle wrote: post #35

This is great for internal distributed systems where having ordered keys is useful, however, it should probably be noted that these probably shouldn't be used as public identifiers (even though this will probably be the defacto standard and used publicly without thought). Having any information, specifically time information, leaking from your systems may or may not have unanticipated security or business implication…

Thinking that harder-to-guess IDs will mitigate attacks is an example of security by obscurity. It's better to think of any IDs in your database as being public knowledge, because they will leak anyway. Assuming that no one can guess another ID leads to shoddy practices. I generally keep IDs sequential and build security around the basic assumption that IDs are not keys, passwords, sessions, or secrets - they're just…

Doesn't that still leak (statistical) information?

It may not be technically security, but e.g. knowing your competitor just added N products to their shop, might be a security issue for the business.

Re: Goodbye integers, hello UUIDv7

#102

Post by throwawayb4dc65 »

Traubenfuchs wrote: post #71

Am I the only one instinctively upset by the communication/bandwith/storage overhead of the dashes as well as the version and variant bits of UUIDs? It might be insignificant, but to me it makes UUID feel tainted, dirty. 11.1% of a UUID are dashes. 15.3% of a UUID are wasted bits if you count version and variant bits. Anecdote: I worked for a company that used numeric primary ids internally and externally and increas…

The dashes are just there to help separate the groups visually. You can write code to remove/add the dashes if you want to shorten the URL. If you use the uuid data type when storing it in a database, it only uses 128 bits.

Re: Goodbye integers, hello UUIDv7

#103

Post by woile »

jonhohle wrote: post #35

This is great for internal distributed systems where having ordered keys is useful, however, it should probably be noted that these probably shouldn't be used as public identifiers (even though this will probably be the defacto standard and used publicly without thought). Having any information, specifically time information, leaking from your systems may or may not have unanticipated security or business implication…

Could you explain a bit more how it would be a risk? Maybe for session tokens is understandable. But why leaking account created info is a problem?

Re: Goodbye integers, hello UUIDv7

#104

Post by miiiiiike »

This is neat. I've been using a custom snowflake cluster for years. Having this in the language/DB would be great for smaller projects.

For bigger/public projects I'd like to be able to add a sequence, node, and data center id to the UUID too.

Re: Goodbye integers, hello UUIDv7

#105

Post by CSDude »

tgv wrote: post #70
andrewmatte wrote: post #51

Earlier quoted context omitted.

jonhohle, thanks. Do you know of examples of when milliseconds are part of the session tokens or accounts being created has been exploited?

The German tank production capacity was estimated by serial numbers of captured tanks. There are ways to read all kinds of information by observing energy usage. High resolution time and sequence data undoubtedly reveal more than you’d like.

Most of our lives as boring SaaS etc. software developer will not be near as exciting as this, but of course you may never know.

I parsed the EV chargers APIs where I live (using Frida in Android) and one of the fields returned the daily revenue and profit.

Re: Goodbye integers, hello UUIDv7

#106

Post by [deleted] »

tzahifadida wrote: post #94

To me it sounds like a corner case. Example: a) UUID4, CreatedTime/UpdatedTime. b) Bigint, CreatedTime/UpdatedTime. c) UUID7 internal (which also includes time badly), UUID4 external/whatever short ID. How exactly this helps if you need external ids (which you usually do today)? It doesn't even make it a short ID. Even if there is a corner case, are we just saving a few bytes while adding more complication? Clustered…

[deleted]

Re: Goodbye integers, hello UUIDv7

#107

Post by hughesjj »

berkes wrote: post #101
noduerme wrote: post #92

Earlier quoted context omitted.

Thinking that harder-to-guess IDs will mitigate attacks is an example of security by obscurity. It's better to think of any IDs in your database as being public knowledge, because they will leak anyway. Assuming that no one can guess another ID leads to shoddy practices. I generally keep IDs sequential and build security around the basic assumption that IDs are not keys, passwords, sessions, or secrets - they're just…

Doesn't that still leak (statistical) information? It may not be technically security, but e.g. knowing your competitor just added N products to their shop, might be a security issue for the business.

You're absolutely right, this is also why you generally encrypt sessionized or "consistent view" pagination tokens for public apis (save for primitives like ddb or Kafka)

The end user should know no details about your internal key space.

Re: Goodbye integers, hello UUIDv7

#108

Post by jillesvangurp »

rockwotj wrote: post #3

I find it interesting that it’s quoted random IDs are bad for performance, because it’s actually better for distributed storage systems because you don’t hotspot on a single node. For example see: https://stackoverflow.com/a/53901549 and https://medium.com/google-cloud/cloud-spanner-choosing-the-r...

Depends how you query it. In a lot of systems, recently added data is also the most queried data or data typically gets pulled out sorted by time. Having that data on disk in more or less the order it is going to be queried makes sorting it a bit easier. Even in a sharded system, each of the shards would have less work to do for sorting. Of course a lot of these systems would have an append only write model which would effectively sort things by time anyway, even with completely random ids.

Somebody posted an interesting article for the instagram ids, which do something similar. They use 41 bits for a time from a custom epoch followed two more groups of bits for a shard id and a sequential number. Each shard has an incrementing sequence for the sequential bit, which guarantees that things on a shard are sorted by time.

This UUIDv7 is slightly weaker than that but sorting things published in the same millisecond is mostly going to be very light work. The lack of a dedicated sharding group of bits is not that important as you could just take the n least significant bits at the end for that without too much effort. Those are random so you end up with nice consistent hashing. Having 48 instead of 41 bits for the time means we won't run out of time any time soon (nearly 9K years vs. 70 years).

Re: Goodbye integers, hello UUIDv7

#109

Post by hinkley »

chacham15 wrote: post #9
rockwotj wrote: post #3

I find it interesting that it’s quoted random IDs are bad for performance, because it’s actually better for distributed storage systems because you don’t hotspot on a single node. For example see: https://stackoverflow.com/a/53901549 and https://medium.com/google-cloud/cloud-spanner-choosing-the-r...

They can be bad for performance. It all depends on your access patterns. A common caching pattern is called "temporal locality" which means that theres a high likelihood that data created at the same time will be accessed at the same time. Therefore, if these pieces of information are on the same machine, they can be queried / returned much faster than if they were both on separate machines. This is doubly true if th…

If you're working on a multi-user system, particularly one with hundreds of requests per second, there is no locality of ids. Two of my actions are separated by a sea of actions by other users.

Re: Goodbye integers, hello UUIDv7

#110

Post by berkes »

chacham15 wrote: post #9
rockwotj wrote: post #3

I find it interesting that it’s quoted random IDs are bad for performance, because it’s actually better for distributed storage systems because you don’t hotspot on a single node. For example see: https://stackoverflow.com/a/53901549 and https://medium.com/google-cloud/cloud-spanner-choosing-the-r...

They can be bad for performance. It all depends on your access patterns. A common caching pattern is called "temporal locality" which means that theres a high likelihood that data created at the same time will be accessed at the same time. Therefore, if these pieces of information are on the same machine, they can be queried / returned much faster than if they were both on separate machines. This is doubly true if th…

We solved that with UUIDS and updated_at and created_at columns. The latter the default sort in all views and queries. So the btree/indexing issues were hardly an issue. Whenever you fetch a set of rows, they will be bounded by these timestamps.

We even sharded on these columns, because of this (our business case made it so that hardly ever did people need data over multiple months)

But we never encountered distribution issues. I don't think the locality issue will be solved, as postgres doesn't consider other columns when distributing data, only the primary key IIRC. I don't know why we never saw this, though.

Post reply on HN