41–50 of 54 posts
Yes, but that's for 1,000 IDs per hour. That is not a large workload. If you generate 10 IDs per second (which, again, is not a lot), your time frame shrinks to just over 1 year. With 100 IDs per second, a mere 36 days. With your reduced alphabet, I would at the very least bump your ID length to 16 characters.
Kind of curious as to why they publish this. It's interesting, but if I did think I would just see it as an implementation detail and not something particularly worthy of an article. Should I maybe change my standards for what's publishable?
If you want pseudo-random counters for something that are guaranteed to not have collisions, consider using a "linear feedback shift register". LFSRs allow you to choose the number of bits in your id, and "complete" LFSRs use a starting seed [+] value that guarantee they will exhaust the entire bit space before repeating. They are very cool. In distributed environments, you can assign different seeds to individual no…
In order to attain that property the seed must be kept secret, right? A secret that cannot be rotated is hard to work with.
> This gives us a 1% probability of a collision in the next ~35 years if we are generating 1,000 IDs per hour. Yes, but that's for 1,000 IDs per hour . That is not a large workload. If you generate 10 IDs per second (which, again, is not a lot), your time frame shrinks to just over 1 year. With 100 IDs per second, a mere 36 days . With your reduced alphabet, I would at the very least bump your ID length to 16 charact…
They have a problem (long IDs in links), they have a scenario (no more than 1000 entities, databases, I suppose, created every hour), and they found something simple that solves their problem under the requirements derived from their scenario. YMMV
If you want pseudo-random counters for something that are guaranteed to not have collisions, consider using a "linear feedback shift register". LFSRs allow you to choose the number of bits in your id, and "complete" LFSRs use a starting seed [+] value that guarantee they will exhaust the entire bit space before repeating. They are very cool. In distributed environments, you can assign different seeds to individual no…
From TFA: > We knew that we wanted to avoid using integer IDs so that we wouldn’t reveal the count of records in all our table In order to attain that property the seed must be kept secret, right? A secret that cannot be rotated is hard to work with.
But in the case of the "taps", yes, you would want to treat those as a secret if you didn't want someone to predict the next value in the cycle. An LFSR by itself might not be a complete solution for IDs for public consumption, but it does neatly solve the need for pseudo-random non-colliding IDs. Whereas NanoIDs are able to generate values with a ~1% probability of collision over some time period, LFSRs can generate values with a 0% probability of collision over their entire bit space. (And they're composable!)
It looks like NanoIDs are entirely random values reduced to a specific alphabet (https://github.com/ai/nanoid/blob/main/index.js) -- which is approximately similar to:
tr -dc _A-Z-a-z-0-9 So if you want randomness in your IDs and you still also want to guarantee 0 collisions, then you can always just append a randomly-generated value to the output of the LFSR, and then feed that into whatever widget you want to use to convert it into a public ID, if you think it's necessary.
But the edge case of collision prevention at really large scales has always seemed to me to be much harder to solve than the edge case of ID enumeration giving up anything valuable (in a well-designed application).
We should consider the following properties when evaluating ID formats and generation algorithms: 1. Private: you shouldn’t be able to gain information about the system using the IDs from an ID alone. E.g. document enumeration attacks like what happened with Parler ( https://www.wired.com/story/parler-hack-data-public-posts-im... ) 2. B-tree/cache friendly: newly created IDs should all exist in a narrow range of valu…
looks like there's an impl already: https://github.com/cbschuld/uuid-base58
I wonder how this might compare to just storing regular autoincrementing ints in the database, and converting to/from hashids ( https://hashids.org/ ) at the edge. It eliminates the collision concern and stores more compactly at the cost of a tiny amount of encode/decode when processing requests. You’d want to push it down as close to the database layer as possible to avoid inadvertent int ID leaks; I added native ha…
Those are not cryptographically secure. It would not be hard for someone to figure out how to decode it.
After reading this article, my impression of Planetscale as a brand actually got worse. It seem to miss most of the essential bits of information: * The basic concept is that they just want to use a bigger alphabet to encode more information in fewer characters. The efficiency ratio of NanoID is log 36/log 16, or ~30% better since it has a bigger alphabet. You could get more if you went for instance with base 58 (inc…