Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sqids – Generate short unique IDs from numbers

sqids.org

Post reply on HN
Print view View original

131–140 of 249 posts

Re: Sqids – Generate short unique IDs from numbers

#131

Post by infogulch »

leobg wrote: post #114
pacificmint wrote: post #77

Earlier quoted context omitted.

> you can tell the growth rate of the company. You can even do this when you don’t know the exact interval by using probabilities. The Allies used this method to estimate German tank production in World War II by analyzing the serial numbers of captured or destroyed tanks. This is know as the German Tank Problem [1] [1] https://en.wikipedia.org/wiki/German_tank_problem

Very interesting. I’m a lawyer and using sequential IDs in a fraud case right now, to determine the number of victims. Unfortunately, so far, I only have the IDs of two victims, and those are from just within about a month, whereas the fraud has likely been going on for several years. Just simply extrapolating that growth rate isn’t going to be very accurate. Also, I suspect that the perpetrators did not start at ID…

You might try to use the information to find more victims first.

Re: Sqids – Generate short unique IDs from numbers

#132

Post by stavros »

andix wrote: post #121

Actually I'm a bit disappointed that it can't format 128 bit integers or byte arrays. That would allow formatting UUIDs. I'm not a huge fan of public facing integer IDs. There is always the risk of leaking some kind of critical information with ascending IDs. So I will probably keep Base64URL formatting my UUIDs to make them shorter for URLs, QR Codes and so on. Quick example: 20b30b32-d421-4cfb-bdbc-9a4e0475abea =>…

The shortuuid library can do that:

https://pypi.org/project/shortuuid/

Re: Sqids – Generate short unique IDs from numbers

#133

Post by wslh »

I offered something similar here [1] and it is used by many companies including Philip Morris, and the Argentinian tax agency for the same purposes.

The technique I used (I should publish it as open source) is using a Feistel cipher [2] with a key. The Feistel network could be adjusted to almost any size and the key used in every round is an expansion of a general key using a key derivation function [3] (KDF3 if I remember well).

Basically it is a symmetric cipher of arbitrary size.

[1] https://www.nektra.com/products/secure-coupon-code-generator...

[2] https://en.wikipedia.org/wiki/Feistel_cipher

[3] https://en.wikipedia.org/wiki/Key_derivation_function

Re: Sqids – Generate short unique IDs from numbers

#134

Post by hinkley »

parhamn wrote: post #70

Side note: there are some business insights you can get from a company using serial ids. i.e if you sign up and get user id 32588 and make another account a few days later, you can tell the growth rate of the company. And this is possible with every resource type in the application. I do wonder how much the url bar junk thing matters these days. I tend to use uulids (waiting on uuid v7 wide adoption), and they're a b…

And this is the stuff you get if you manage to get your access control right.

Get it wrong and we jump from actionable business metadata to actionable business data (like perhaps which of your customer's customers are poachable)

Re: Sqids – Generate short unique IDs from numbers

#135

Post by jl6 »

Sharlin wrote: post #74
njharman wrote: post #53

Earlier quoted context omitted.

> That doesn't seem possible. How would that work? agree; b00b, DlCK, cntfcker But I suppose, if user doesn't get to craft input, the collision space of converted numerical ids and words like above is sufficiently small to be ignorable.

Besides vowels, nanoid excludes 0, 1, 3, 4, 5, I, l, x, X, v, V, and other lookalikes, so the chances of generating something naughty in any language are close to zero.

Humans have a high capacity for spotting rudeness. Nanoid’s nolookalikesSafe alphabet would allow blwjb69FKmyD7CK.

(Sorry)

Re: Sqids – Generate short unique IDs from numbers

#137

Post by hinkley »

runlevel1 wrote: post #51
habitue wrote: post #33

Skipping profanity seems like a liability in this design. It means in order to preserve the encoding you need to make the banned word list immutable, otherwise old sqids will decode to the wrong thing when you get them back.

The stupid simple way I did this ages ago was: 1. Start with a-z. 2. Drop all vowels, numbers, most homoglyphs, and the letter 'x'. 3. Map digits 0-9 to one of the remaining letters. 4. Stringify the integer and replace the digit in each decimal place with its corresponding character. For my use-case, all the numbers were >7 digits long, so the odds of you getting an offensive acronym were reasonably low unless you s…

Most numbers can be used as letters or phonemes.

I could give a fuck about avoiding swear words, but if you want to avoid slurs and eyebleach-inducing ideas and still have any sort of compact representation, I suspect we have to look not at problematic letters but problematic groups of letters. There's nothing intrinsically wrong with the letter E. Not with G, I, N, or R, but you can sure get a lot of attention you don't want by arranging them in the wrong order. K and Y aren't bad either, unless you're hating on Jewish people.

So maybe there's a 5:4 or a 5:3 encoding out there where you avoid making syllables.

Re: Sqids – Generate short unique IDs from numbers

#138

Post by erhaetherth »

filleokus wrote: post #104
8n4vidtmkvmk wrote: post #95

Earlier quoted context omitted.

Add an offset, multiply by a large prime number, and modulo. I don't think you can recover the original number without figuring out the prime.

Ah, that's neat. Why is the offset necessary?

Might not be, but I like to start with a big number instead of 0 or 1 to fill all the bits. For example, if your prime is 100019, then your first number in binary is 00011000011010110011 but if your max number is something like 2^53 (00100000000000000000000000000000000000000000000000000000) then you have a lot of unfilled bits. The way I have mine set up, the output ID is always exactly 9 chars. 99% of the time it's naturally 9 chars because just by probability most of the numbers will be large, but some of them come out 8 chars and then I just pad up to 9 so it's nice and consistent.

Re: Sqids – Generate short unique IDs from numbers

#139

Post by wslh »

waffle_ss wrote: post #57

I wrote a Ruby gem to address this problem of hiding sequential primary keys that uses a Feistel network to effectively shuffle int64 IDs: https://github.com/abevoelker/gfc64 So instead of /customers/1 /customers/2 You'll get something like /customers/4552956331295818987 /customers/3833777695217202560 Kinda similar idea to this library but you're encoding from an integer to another integer (i.e. it's format-preservin…

I recommend to review my comment where I also use a Feistel cipher [1] but the difference is that it is not limited to int64 but I can even use 8 bits. Obviously loosing security properties but working as an obfuscation method. If you use a random source with relatively few bits you should check if there are duplicates while with the Feistel cipher you are sure there isn't.

[1] https://news.ycombinator.com/item?id=38418198

Re: Sqids – Generate short unique IDs from numbers

#140

Post by rafram »

8organicbits wrote: post #82
habitue wrote: post #33

Skipping profanity seems like a liability in this design. It means in order to preserve the encoding you need to make the banned word list immutable, otherwise old sqids will decode to the wrong thing when you get them back.

Agreed, this is a big risk made worse that the default word list can change over time. https://sqids.org/faq#future-blocklist

It should at least take a blocklistVersion parameter (or similar), even if passing the wrong version just generates an error.
Post reply on HN