Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

github.com

Post reply on HN
Print view View original

211–220 of 236 posts

Re: Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

#211

Post by kbumsik »

spoiler wrote: post #207
kbumsik wrote: post #151

Earlier quoted context omitted.

> Re: prefix, is the concern that I haven't defined the allowed character set as part of the spec? It would be great if you add suggestions for compound types (like “article-comment”) in README as OP stated as well.

It seems that's not allowed currently, if I'm reading it right. I'm not sure I like `-` very much. The reason why I don't like it is because of how double-click to select and line breaking works for the dash. Maybe allowing `_` in the typename, and the have the rightmost `_` serve as the separator might be more consistent. But also, I'm bike-shedding and its only an ID

I like using "." for this case. Because types definition typically belong to a package or module, which commonly uses "." for separator.

Re: Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

#212

Post by unshavedyak »

fauigerzigerk wrote: post #191
unshavedyak wrote: post #171

Earlier quoted context omitted.

Is there a good reason why Base63 (nopad) doesn't exist? Ie Base64 minus the `-`, so that you almost get the density of base64 (nopad) but the double click friendly feature. I was reviewing encodings recently and didn't want to drop all the way down to base32, but for some reason the library i was using didn't allow anything beyond base32 and bas64 variants, despite having a feature where you can define your own base…

>Is there a good reason why Base63 (nopad) doesn't exist? Ie Base64 minus the `-`, so that you almost get the density of base64 (nopad) but the double click friendly feature. Yes, the reason is that you need 64 characters if you want each character to encode 6 bits as log2(64) == 6. If you only have 63 characters in your alphabet then one of your 6-bit combinations has no character to represent it. Base32 can represe…

Is that "just" a performance concern though? Ie why is there a base58 and base62 but no base63?

Now you've got me curious on the performance of base58 to base64 hah. Down the rabbit hole i go. Appreciate your reply, thanks :)

Re: Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

#213

Post by hot_gril »

lll-o-lll wrote: post #115

K-Sortable is a great concept; having weakly sorted keys solves a bunch of use-cases. I really like the idea of a typed, condensed string representation. However I wonder if an unintended side affect of UUID V7 is going to be a bunch of security problems. People aren’t meant to use uuids as tokens, and they aren’t supposed to use PKs from a DB for this either - but they do. Because UUID v4 is basically crypto random,…

I can see some use cases for it, but every time in the past I've encountered other kinds of partially-sequential UUIDs like v1 or v5, they've been misused. Same with the hash-based ones like v3. v4 is simple and not prone to misuse.

Re: Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

#214

Post by hot_gril »

atonse wrote: post #193
lll-o-lll wrote: post #115

K-Sortable is a great concept; having weakly sorted keys solves a bunch of use-cases. I really like the idea of a typed, condensed string representation. However I wonder if an unintended side affect of UUID V7 is going to be a bunch of security problems. People aren’t meant to use uuids as tokens, and they aren’t supposed to use PKs from a DB for this either - but they do. Because UUID v4 is basically crypto random,…

This brings up an interesting ergonomics problem. By naming them UUIDv4 and UUIDv7, is it going to be this never ending confusion for people to have to remember which one is good for databases and which one good for one time tokens? Not sure what the backwards compatible solution here is either. In elixir the function is UUID.uuid4() to generate a v4 UUID. So we could theoretically scan code for its use I suppose. Bu…

> is it going to be this never ending confusion for people to have to remember which one is good for databases and which one good for one time tokens

Yes, because this is what's been happening already with the past versions. It's not just sequential and random, there are also hash-based UUIDs. They shouldn't have sequential (heh) version numbers.

Re: Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

#215

Post by hot_gril »

carlsverre wrote: post #78
kortex wrote: post #65

Does the prefix ("user_") get recorded in the DB (so every string in the column starts with the same "user_"), or does are there constraints and other clever chicanery to save those bytes in every record? Or do modern DB engines even care? Is this premature optimization?

The authors have created a specialisation for Postgres that leverages a custom type which is a tuple of type and uuidv7: https://github.com/jetpack-io/typeid-sql/blob/main/sql/typei... This is more optimal for Postgres while making it slightly more difficult to interop between the db and the language (db driver needs to handle custom types, and you need to inject a custom type converter). And while there are hacks yo…

In my experience, using just uuid as a pkey in Postgres already causes noticeable slowdowns vs the typical bigint. I wouldn't jump into anything other than bigint pkeys unless I'm solving an existing problem and have benchmarks to prove it.

Re: Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

#216

Post by hot_gril »

armchairhacker wrote: post #142
onlypositive wrote: post #139

> Compare to entirely random global ids, like UUIDv4, that generally suffer from poor database locality. What does this mean in more words?

Because the bytes are all random, UUIDv4 is sorted randomly. So whenever you insert a database entry with a new UUID, it ends up getting put in some random memory location. In practice, you often want to select database entries which were inserted near each other in time. Ex: you would like to select the most recent entries or entries within a time frame. Even when selecting entries by other information, entries inse…

Right, but worth noting that distributed DBs don't necessarily play nice with sequential pkeys. Spanner explicitly tells you not to use a time-based pkey.

Re: Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

#217

Post by yencabulator »

inopinatus wrote: post #159

I'm not wild about the Crockford encoding. In practice I've found it to be a flat-out mistake when you come to provide technical support or analysis for values encoded this way. The Crockford alphabet is based on design goals that are rarely encountered in practice, such as pronouncing identifiers over the phone. It introduces ambiguity, which is a disaster for grepping logs or any other circumstances where you might…

Wait, where's the hyphen in Crockford Base32? https://en.wikipedia.org/wiki/Base32#Crockford's_Base32

My favorite base-32 encoding is z-base-32, which I find just gentler on the eyes: https://philzimmermann.com/docs/human-oriented-base-32-encod...

The biggest problems with base58 are 1) it works for integers, less so for arbitrary binary data like crypto keys 2) case-sensitivity ISnOtNIcEtOLoOKaT (in my opinion).

Re: Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

#218

Post by [deleted] »

dloreto wrote: post #173
inopinatus wrote: post #159

I'm not wild about the Crockford encoding. In practice I've found it to be a flat-out mistake when you come to provide technical support or analysis for values encoded this way. The Crockford alphabet is based on design goals that are rarely encountered in practice, such as pronouncing identifiers over the phone. It introduces ambiguity, which is a disaster for grepping logs or any other circumstances where you might…

I hear you ... and I debated using either base58 or base64url. I do like the more compact encoding they provide. Ultimately I ended up leaning towards a base32 encoding, because I didn't want to pre-suppose case sensitivity. For example, you might want to use the id as a filename, and you might be in an environment where you're stuck with a case insensitive filesystem. Note that TypeID is using the Crockford alphabet…

[deleted]

Re: Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

#219

Post by dloreto »

dolmen wrote: post #199
dloreto wrote: post #134

Earlier quoted context omitted.

Thanks for the feedback! We have tests for the base32 encoding which is the most complicated part of the implementation ( https://github.com/jetpack-io/typeid-go/blob/main/base32/bas... ) but your point stands. We'll add a more rigorous test suite (particularly as the number of implementations across different languages grows, and we want to make sure all the implementations are compatible with each other) Re: prefix…

There is no tests . There is just a single test. Which only tests the decoding of a single known value. No encoding test. Go has infrastructure for benchmarking and fuzzing. Use it! Also, you took code from https://github.com/oklog/ulid/blob/main/ulid.go which has "Copyright 2016 The Oklog Authors" but this is not mentionned in your base32.go.

We've now implemented pretty thorough testing: https://github.com/jetpack-io/typeid-go/blob/main/typeid_tes...

Thanks for the feedback!

Re: Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs

#220

Post by dloreto »

klabb3 wrote: post #120

A couple of suggestions: Lock down the prefix string now before it’s too late and document it. I see in Go that it’s lowercase ascii, which seems fine except for compound types (like “article-comment”). May be worth looking at allowing a single separator given that many complex projects (and ORMs) can’t avoid them. The Go implementation has no tests. This is very unit-testable. Add tests goddammit! For Go, I’d align…

A follow up:

1. We've now implemented pretty thorough testing: https://github.com/jetpack-io/typeid-go/blob/main/typeid_tes...

2. I clarified the prefix in the spec

Thanks for the feedback!

Post reply on HN