Live data from Hacker News

Sqids – Generate short unique IDs from numbers

sqids.org

111–120 of 249 posts

Re: Sqids – Generate short unique IDs from numbers

#111
post #96

How do you adjust or evolve the blocklist with this, without making previously generated IDs incorrect? The ID is simply incremented if it is blacklisted [1]. So the ID is fixed to the blacklist content, and adjusting it in any way invalidates certain segments of previously generated IDs? 1. https://github.com/sqids/sqids-rust/blob/9f987886bc06875d782...

They address it here: https://sqids.org/faq#future-blocklist

You're right that you can't update the blocklist in a backwards compatible way.

Re: Sqids – Generate short unique IDs from numbers

#112

Neat library! We're using randomly generated strings for many things. IDs, password recovery tokens, etc. We've generated millions of them in our system, for various use-cases. Hundreds of thousands of people see them every day. I've never heard any complaints about a random content-id being "lR8vDick4r" (dick) or whatever. But nowadays our society is so afraid of offending anyone, that profanity filters has extended…

The block list is 2/3 of the (minified) library. I found this entire choice odd.

First, it's highly incomplete because you can find at least 10x more combinations spelling the same "word". And probably 10x more slurs that aren't in this block list. Second, because it's hardcoded in your source. Third, because there are more elegant solutions.

Such as to pick an alphabet that can't spell readable words unless you're trying really hard to read a slur into it. Say this (no vowels or digits):

bcdfghjklmnpqrstvwxyzBCDFGHJKLMNPQRSTVWXYZ (length 42)

The full lower+upper+digits alphabet they use is 62. Feels like you're losing a lot, but... not really.

- A 128-bit id in base 62 = 22 letters.

- A 128-bit id in base 42 = 24 letters.

JUST TWO MORE LETTERS. And it's one more letter for 64-bit id (11 vs 12). And we can avoid this entire silliness. The problem is the author doesn't realize that logN is... logarithmic, I suppose.

Re: Sqids – Generate short unique IDs from numbers

#113
post #93

Earlier quoted context omitted.

The stupid simple way I did this ages ago was: 1. Start with a-z. 2. Drop all vowels, numbers, most homoglyphs, and the letter 'x'. 3. Map digits 0-9 to one of the remaining letters. 4. Stringify the integer and replace the digit in each decimal place with its corresponding character. For my use-case, all the numbers were >7 digits long, so the odds of you getting an offensive acronym were reasonably low unless you s…

> California Personalized License Plate Requests Flagged for Review 2015-2016: https://docs.google.com/spreadsheets/d/18IUVU9Q4uN_lxqNd5AsN ... Wow this is a funny peek into a weird perdicment where people need to justify that they have a good reason to have a specific license plate. Some seems obviously ok such as: INT13H 314 PI And some are obviously not: DRY(hand emoji)JOB DICK OUT Come to think of it: Can license…

California allows one hand, star, or heart shape in the plate.

Re: Sqids – Generate short unique IDs from numbers

#114
post #70

Side note: there are some business insights you can get from a company using serial ids. i.e if you sign up and get user id 32588 and make another account a few days later, you can tell the growth rate of the company. And this is possible with every resource type in the application. I do wonder how much the url bar junk thing matters these days. I tend to use uulids (waiting on uuid v7 wide adoption), and they're a b…

> you can tell the growth rate of the company. You can even do this when you don’t know the exact interval by using probabilities. The Allies used this method to estimate German tank production in World War II by analyzing the serial numbers of captured or destroyed tanks. This is know as the German Tank Problem [1] [1] https://en.wikipedia.org/wiki/German_tank_problem

Very interesting.

I’m a lawyer and using sequential IDs in a fraud case right now, to determine the number of victims.

Unfortunately, so far, I only have the IDs of two victims, and those are from just within about a month, whereas the fraud has likely been going on for several years. Just simply extrapolating that growth rate isn’t going to be very accurate.

Also, I suspect that the perpetrators did not start at ID 1.

Re: Sqids – Generate short unique IDs from numbers

#115

I haven’t been able to find a case for this because ids either need to be unique or they’re not going to be large. If they’re unique, I’m using uuid or ulid (uuidv7 of tomorrow) as the sortable primary key type to avoid conflicts without using the db to generate and maintain sequences. Where do you have unique ids that aren’t the primary key? I would be more interested in a retrospectively unique truncated encoding f…

I’m currently using sqids as slugs to have a shorter url than just using my uuid primary key

Re: Sqids – Generate short unique IDs from numbers

#116
This looks handy.

Given the retry-on-bad-word feature, I was sceptical of the no-collision claim -- but after looking at the JS source code, I'm confident it's correct.

For each number encoded, one character from the alphabet is "held back" to use as a delimiter (with the particular character chosen changing as each number is processed, presumably to make the output "look more random"). The very first character output is essentially a "free choice" that selects the initial permutation of the alphabet to use.

The algorithm is implemented as a function encodeNumbers() that calls toId() to encode each number, then checks for bad words and recurses with a new value of "increment" if it finds any. To prove correctness it's helpful to imagine an "in-between" function, tryEncodeNumbers(numbers, alphabet), which the outer encodeNumbers() calls in a loop to do most of its work (pseudocode):

  function encodeNumbers(numbers) {
    offset = sum(numbers)
    do {
      result =  tryEncodeNumbers(numbers, permute(alphabet, offset + increment++))
    } while (badWordIn(result))
    return result
  }
Here permute() is a function that permutes the characters in its first (string) argument according to its second (integer) argument in some arbitrary way.

The toId(num, alphabet) function, which simply encodes a single number using "digits" taken from the characters in the alphabet parameter, is clearly injective with respect to the num parameter provided that alphabet contains no duplicate characters -- that is, if we hold some duplicate-free alphabet string fixed, every distinct value of num produces a distinct encoded string as output. (For example, toId(42, "0123456789") gives "42", and no other value of num produces this string when the alphabet remains unchanged.) tryEncodeNumbers(numbers, alphabet) first outputs a character representing alphabet (i.e., its initial alphabet -- which is its complete internal state), then joins together a bunch of these toId()-encoded numbers, with an extra character in between each that is known not to appear as a digit in the preceding number, permuting the alphabet in an alphabet-dependent but num-independent way for each encoded number output. Because the alphabet permutation is independent of the input array of numbers, this means that the alphabet used to encode the i-th number depends only on the initial alphabet and i. This means that, again holding its initial alphabet fixed, tryEncodeNumbers() is likewise injective with respect to the input array of numbers. (Suppose it were not: Then there is an initial dupe-free alphabet, and two distinct arrays of numbers, that produce the same output. Find the first position where the two arrays differ. Since the final results are identical by assumption, one of the two encoded outputs for this position must be a prefix of the other. But if the two encoded outputs are of different lengths, the shorter one must either terminate the entire string, making it shorter than the other encoded string, or be immediately followed by a character that cannot appear in the output of toId() with the alphabet used at this position, both of which contradict the assumption that the resulting strings are equal. Therefore toId() must have output identical strings for the two different numbers at this position, when given the same alphabet. But this contradicts injectivity of toId(), so this (non-injectivity of tryEncodeNumbers()) is impossible.)

The final step is to see that, if encoding two inputs with the top-level encodeNumbers() function gives the same first character C, it must be because the first successful (bad-word-free) loop iteration for the first input passed the same alphabet to tryEncodeNumbers() as the first successful loop iteration for the second input. If the remainders of the two encoded strings are also equal, then by injectivity of tryEncodeNumbers() for fixed alphabet choice their input number arrays must have also been the same. Since no restrictions were placed on the two inputs, this holds for all possible input pairs -- that is, it is impossible for encodeNumbers() to produce the same encoded output string for two different inputs.

Re: Sqids – Generate short unique IDs from numbers

#118

Neat library! We're using randomly generated strings for many things. IDs, password recovery tokens, etc. We've generated millions of them in our system, for various use-cases. Hundreds of thousands of people see them every day. I've never heard any complaints about a random content-id being "lR8vDick4r" (dick) or whatever. But nowadays our society is so afraid of offending anyone, that profanity filters has extended…

There's other "general" rules when it comes to random human-readable tokens such as not using Os and Is if your strings include numbers - people can and will confuse them with 0s and 1s if they have to type them over.

Most gift card tokens for example don't allow the use of those two (or quietly correct it) to avoid making that mistake.

Re: Sqids – Generate short unique IDs from numbers

#119

I see many people in this thread saying that this is a good way to hide insights from ids/numbers, I don't understand, aren't the generated values easily decoded? couldn't I just decode a couple of numbers to get that insight? What am I missing.

The docs state:

  Not Good For:  
  [...]  
  User IDs  
    Can be decoded, revealing user count
So yeah, just using a sequential id and encoding the number with this library is not a viable idea if you want to hide your insights.

Re: Sqids – Generate short unique IDs from numbers

#120

Neat library! We're using randomly generated strings for many things. IDs, password recovery tokens, etc. We've generated millions of them in our system, for various use-cases. Hundreds of thousands of people see them every day. I've never heard any complaints about a random content-id being "lR8vDick4r" (dick) or whatever. But nowadays our society is so afraid of offending anyone, that profanity filters has extended…

The block list is 2/3 of the (minified) library. I found this entire choice odd. First, it's highly incomplete because you can find at least 10x more combinations spelling the same "word". And probably 10x more slurs that aren't in this block list. Second, because it's hardcoded in your source. Third, because there are more elegant solutions. Such as to pick an alphabet that can't spell readable words unless you're t…

Totally agree. 2/3 is wild, especially given it seems like you could mitigate most of the risk just by removing vowels from the dictionary.
Post reply on HN