Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sqids – Generate short unique IDs from numbers

sqids.org

Post reply on HN
Print view View original

51–60 of 249 posts

Re: Sqids – Generate short unique IDs from numbers

#51

Post by runlevel1 »

habitue wrote: post #33

Skipping profanity seems like a liability in this design. It means in order to preserve the encoding you need to make the banned word list immutable, otherwise old sqids will decode to the wrong thing when you get them back.

The stupid simple way I did this ages ago was:

1. Start with a-z.

2. Drop all vowels, numbers, most homoglyphs, and the letter 'x'.

3. Map digits 0-9 to one of the remaining letters.

4. Stringify the integer and replace the digit in each decimal place with its corresponding character.

For my use-case, all the numbers were >7 digits long, so the odds of you getting an offensive acronym were reasonably low unless you started combining them.

But there's no perfect solution. As this dataset shows, you can find offense in almost anything if you look hard enough:

California Personalized License Plate Requests Flagged for Review 2015-2016: https://docs.google.com/spreadsheets/d/18IUVU9Q4uN_lxqNd5AsN...

Re: Sqids – Generate short unique IDs from numbers

#52

Post by exxos »

progne wrote: post #38

In a Ruby app we just convert to a high base, like > 1234567890.to_s(36) => "kf12oi" That gets us most of the way there, but Sqid has a Ruby library and lets you set a much higher base, including upper case characters, and I suppose, emoji. We're going to need much bigger numbers before that space savings makes much difference. I like it, but it's hard to know when something like that is worth adding a dependency.

I didn't think of that, but this is a nice trick!

Re: Sqids – Generate short unique IDs from numbers

#53

Post by njharman »

lxgr wrote: post #12
no_wizard wrote: post #9

I like the idea, though I use nanoid with the safe letter dictionary (it excludes letters used for profanity[0]) They should use a similar dictionary approach IMO because I looked at the implementation and it’s hardcoded to look for “bad” words Otherwise looks real straightforward! I’d love to see some performance test suites for it [0]: https://github.com/sqids/sqids-javascript/blob/ebca95e114932... [1]: though with…

> it excludes letters used for profanity That doesn't seem possible. How would that work? > I looked at the implementation and it’s hardcoded to look for “bad” words. If you mean https://github.com/y-gagar1n/nanoid-good , that seems to be doing the same thing. In general, I'm a bit weary of solutions that "guarantee no bad words" – this is usually highly language-specific: One language's perfectly acceptable name is…

> That doesn't seem possible. How would that work?

agree; b00b, DlCK, cntfcker

But I suppose, if user doesn't get to craft input, the collision space of converted numerical ids and words like above is sufficiently small to be ignorable.

Re: Sqids – Generate short unique IDs from numbers

#54

Post by wombatpm »

echelon wrote: post #47
vyrotek wrote: post #42

Earlier quoted context omitted.

I believe a big part of the idea is for the hash to be unpredictable as well. If I figure out you're using (36) then I know the next number 1234567891 is "kf12oj". Not the case with Sqids.

I'd prefer to use crockford-encoded entropy with Stripe-style token prefixes to create unique ID namespaces. Run in through a bad words filter, and it's perfect. user_1hrpt0xpax7ps file_xpax7psaz0tv6az0tv6 Etc. In distributed systems you can use the trailing bytes to encode things like author cluster, in case you're active-active and need to route subsequent writes before create event replication. Easy to copy, debug…

Yeah don’t forget the bad words filter. I worked on an IKEA mailing where the list processing house was adding an autogenerated discount code to the address label. The customers received codes with BOOB, DICK, TWAT, and CUNT embedded within. People were not happy.

Re: Sqids – Generate short unique IDs from numbers

#55

Post by [deleted] »

vyrotek wrote: post #42
progne wrote: post #38

In a Ruby app we just convert to a high base, like > 1234567890.to_s(36) => "kf12oi" That gets us most of the way there, but Sqid has a Ruby library and lets you set a much higher base, including upper case characters, and I suppose, emoji. We're going to need much bigger numbers before that space savings makes much difference. I like it, but it's hard to know when something like that is worth adding a dependency.

I believe a big part of the idea is for the hash to be unpredictable as well. If I figure out you're using (36) then I know the next number 1234567891 is "kf12oj". Not the case with Sqids.

[deleted]

Re: Sqids – Generate short unique IDs from numbers

#56

Post by ComputerGuru »

I haven’t been able to find a case for this because ids either need to be unique or they’re not going to be large. If they’re unique, I’m using uuid or ulid (uuidv7 of tomorrow) as the sortable primary key type to avoid conflicts without using the db to generate and maintain sequences.

Where do you have unique ids that aren’t the primary key? I would be more interested in a retrospectively unique truncated encoding for extant ulid/uuid; ie given that we’ve passed timestamp foo, we know that (where no external data is merged) we only need a bucketed time granularity of x for the random component of the id to remain unique (for when sortability is no longer needed).

Or just more generally a way to convert a ulid/uuidv7 to a shorter sequence if we are using it for external hash table lookups only and can do without the timestamp component.

Re: Sqids – Generate short unique IDs from numbers

#57

Post by waffle_ss »

I wrote a Ruby gem to address this problem of hiding sequential primary keys that uses a Feistel network to effectively shuffle int64 IDs: https://github.com/abevoelker/gfc64

So instead of

    /customers/1
    
    /customers/2
You'll get something like

    /customers/4552956331295818987
    
    /customers/3833777695217202560
Kinda similar idea to this library but you're encoding from an integer to another integer (i.e. it's format-preserving encryption). I like keeping the IDs as integers without having to reach for e.g. UUIDs

Re: Sqids – Generate short unique IDs from numbers

#60

Post by 3cats-in-a-coat »

tinco wrote: post #41
3cats-in-a-coat wrote: post #30

I don't get it, that's like two lines of code, why does it have a library and even a domain

[flagged]

I have over 20 years of experience, I run a team, and I can code something like this in my sleep, as well as any average developer (but maybe "average" is lower than where it used to be in my day). Don't talk down to strangers on the Internet so arrogantly, you only end up embarrassing yourself.

I went to check what this code does (beyond a base conversion with a custom alphabet) and I laughed my ass off.

1. Literally TWO-THIRDS of the "library" is a clumsily put together ban list of slurs. Hardcoded, in the library. This DOES NOT belong there.

2. It also contains alphabet checks like say repeated letters, which it'll run every single time it produces an ID, instead of statically checking the alphabet ahead of time, and just using it in production... or even simply selecting an URL-safe alphabet and sticking with it.

3. Despite all the checks, block lists, the default alphabet doesn't do the most elementary thing when producing alpha-numeric ids: exclude letters and digits which look similar, and so will look identical in some fonts. Nasty when you have to write an ID from say a screenshot, or a poster, or what have you.

All in all, that's only borderline more utility than the infamous "leftpad" library, and it may actually be harmful in practice, because it contains a bunch of specific and/or superfluous concerns by the author, almost as if they're trying to justify why this is a library in the first place, but which will likely not align with the precise requirements of the users and their project business rules, while it excludes others (like the one I mentioned in point 3 up there). But I'm sure the kids will be impressed.

Post reply on HN