Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We chose NanoIDs for PlanetScale’s API

planetscale.com

Post reply on HN
Print view View original

11–20 of 54 posts

Re: We chose NanoIDs for PlanetScale’s API

#11

Post by wereHamster »

gobengo wrote: post #6
Jarwain wrote: post #5

I'm curious as to why they didn't decide on using UUIDs with the `-` stripped for URLs, and readding them for queries. It would accomplish the same goals wouldn't it?

NanoIDs are shorter than UUIDs because they have a bigger alphabet. That's probably why. It's not very important for putting IDs in URL paths. But it can matter when using them in DNS subdomains because often there are surprisingly short max character lengths for domain names in e.g. LetsEncrypt SSL certs (iirc 63 characters)

The 63 character limit is in the DNS spec, not invented by Let's Encrypt.

See https://stackoverflow.com/questions/32290167/what-is-the-max... for one explanation.

Re: We chose NanoIDs for PlanetScale’s API

#12

Post by throwaway2016a »

vlmutolo wrote: post #9

We should consider the following properties when evaluating ID formats and generation algorithms: 1. Private: you shouldn’t be able to gain information about the system using the IDs from an ID alone. E.g. document enumeration attacks like what happened with Parler ( https://www.wired.com/story/parler-hack-data-public-posts-im... ) 2. B-tree/cache friendly: newly created IDs should all exist in a narrow range of valu…

It took me a bit to realize that what you're referring to as "error code" in many other contexts is "checksum"... this may just be the domain I work in but for me, "error code" usually means an "id that signifies a specific error happened"

Another thought is that UUID V4 (which is the most common UUID implementation I've seen in the wild) also does not have a checksum. Though I think in general a checksum is probably a good idea if you want to help prevent people from mistyping the code or an optical reader from making an error.

Re: We chose NanoIDs for PlanetScale’s API

#13

Post by nickvanw »

skeeter2020 wrote: post #3

"User friendly, clickable in the browser, api urls" seems like... a made-up user requirement? Have they nailed their product so well that this is the highest value work outstanding? There's some neat technical details in there but to me this is the definition of engineering procrastination.

Hi! I'm the VP of Engineering at PlanetScale and was at the company when we made this decision.

We had the advantage of being able to decide to do this up front when we were first building the current product. The marginal work for this was nearly zero because we did not have to migrate anything existing, we just decided to address our resources using this strategy. In that way, there was nothing procrastinated -we just decided this was a good idea, and did it.

I can speak from firsthand experience, however, how useful it is to be able to copy/paste links and references to resources. It makes support, collaboration and work much easier when you can easily share links to resources.

Re: We chose NanoIDs for PlanetScale’s API

#14

Post by vlmutolo »

throwaway2016a wrote: post #12
vlmutolo wrote: post #9

We should consider the following properties when evaluating ID formats and generation algorithms: 1. Private: you shouldn’t be able to gain information about the system using the IDs from an ID alone. E.g. document enumeration attacks like what happened with Parler ( https://www.wired.com/story/parler-hack-data-public-posts-im... ) 2. B-tree/cache friendly: newly created IDs should all exist in a narrow range of valu…

It took me a bit to realize that what you're referring to as "error code" in many other contexts is "checksum"... this may just be the domain I work in but for me, "error code" usually means an "id that signifies a specific error happened" Another thought is that UUID V4 (which is the most common UUID implementation I've seen in the wild) also does not have a checksum. Though I think in general a checksum is probably…

Good point. “Checksum” is probably the more common word here. I updated my comment.

Re: We chose NanoIDs for PlanetScale’s API

#15

Post by FrancoisBosun »

janus wrote: post #8

This is great, except for the additional query each time a record is instantiated in the rails concern. That might cause some performance problems in high traffic. Perhaps it’d be better to attempt the insertion and change the id only if there’s a colission detected with a uniqueness constraint

In PostgreSQL, if a DB exception is raised (unique_violation), the whole transaction will abort: there is no way to retry.

Re: We chose NanoIDs for PlanetScale’s API

#16

Post by irq-1 »

vlmutolo wrote: post #9

We should consider the following properties when evaluating ID formats and generation algorithms: 1. Private: you shouldn’t be able to gain information about the system using the IDs from an ID alone. E.g. document enumeration attacks like what happened with Parler ( https://www.wired.com/story/parler-hack-data-public-posts-im... ) 2. B-tree/cache friendly: newly created IDs should all exist in a narrow range of valu…

I want a set of symbols which are commonly used in most native languages and easily accessible from the keyboard. Numbers and math/common symbols should be mostly universal. @ from email. Plus, minus, multiply. Less and greater and HTML brackets. Hex without the english characters, and works in URLs.

    0123456789@+-*

Re: We chose NanoIDs for PlanetScale’s API

#19

Post by mnutt »

I wonder how this might compare to just storing regular autoincrementing ints in the database, and converting to/from hashids (https://hashids.org/) at the edge. It eliminates the collision concern and stores more compactly at the cost of a tiny amount of encode/decode when processing requests. You’d want to push it down as close to the database layer as possible to avoid inadvertent int ID leaks; I added native hashids support to clickhouse but I’m not sure what other database support might entail.

Re: We chose NanoIDs for PlanetScale’s API

#20

Post by thaumaturgy »

If you want pseudo-random counters for something that are guaranteed to not have collisions, consider using a "linear feedback shift register". LFSRs allow you to choose the number of bits in your id, and "complete" LFSRs use a starting seed [+] value that guarantee they will exhaust the entire bit space before repeating. They are very cool.

In distributed environments, you can assign different seeds to individual nodes, include the seed in the id, and guarantee no collisions across your entire network.

[+]: edit: sorry, I meant "taps". It's been a minute since I got to do something new with an LFSR. LFSR output is determined by "taps" & "seed" & algorithm. Galois LFSR is an easy algorithm to implement. There are publicly available references and datasets for full-cycle LFSR taps for different bit sizes.

I have a php implementation at https://github.com/robsheldon/asinius-lfsr, but the code is absurdly simple and trivial to translate into any other language. Some references and further notes are included.

Post reply on HN