121–130 of 182 posts
Earlier quoted context omitted.
But now you’ve tried code complexity for a few bytes if storage. That’s just not worth it.
This is why we need 2TB drives now, when we used to get by with 2GB.
Get off my lawn...
But seriously, UUIDs work, they don't need application code to avoid collisions. If you want something a bit more compact/shardable, use ULIDs.
Earlier quoted context omitted.
> if you need a unique ID for an entity with not a lot of records planned (≤10,000,000), why not use a random int64 with a simple for loop on the application side to catch the occasional collisions? What’s the use case for this where UUIDv4 or sequential ID isn’t better? Because it sounds like a solution in search of a problem. > Are there any downsides besides making the application side a tiny bit more complex? Are…
UUIDv4 has performance implications. But I agree, if you are already coupled to the DB (due to the check loop), generally you might as well use sequential. Seems too easy to screw up.
If you have 10 million rows, you're looking at 16MB for the storage of a UUID, vs 8MB for storage of a 64 bit int.
Both of those are entirely cacheable.
Earlier quoted context omitted.
I don't think a lot of the argument that integer IDs reveal too much. Yes, they are guessable but your application should not rely solely on the "secrecy" of the ID to authorize access to a record. If you are worried about someone crawling your public API with wget or curl and an incrementing counter you should re-think whether your data are really public or not, or maybe rate-limit anonymous users, etc. They also re…
> Yes, they are guessable but your application should not rely solely on the "secrecy" of the ID to authorize access to a record Any information you give to a potentially malicious actor can help them attack you. If you have a choice between leaking information and not leaking information, I can’t imagine why you would ever intentionally go with the former, unless you didn’t actually have a choice (feasibility, etc.)…
For example, if some customer ID is used by customers in communication e.g. when calling you on the phone over some billing issue, then there would be strong advantages if your IDs aren't unnecessarily long and if they include some explicit redundancy (e.g. a check-digit with Luhn formula) to protect against communication mistakes.
For another example, if your IDs aren't visible to outsiders but are used in your internal business processes then it may be quite valuable to ensure that IDs of different types (e.g. customer ID vs account ID vs transaction ID) are obviously distinguishable in some way so that someone seeing XXXXXXX knows that it likely is a transaction ID and definitely can't be a customer ID; and it's quite valuable to ensure that you can't have accidental collisions where the same number is a valid ID in different key tables so a bug or miscommunication that confuses them would result in data corruption or information disclosure instead of simply failing.
So "ID design" deserves some attention from UX perspective and blocks of random data aren't optimal UX.
Recently I built a new system (typical business-type backend) and forced to use sqlite + C# + dapper. Using this combination I cannot use guid/uuid as dapper cannot properly map it back to c# from sqlite, and my dislike of int's got me thinking. I have a random string generator (have used it for years for things like OTP's and other reference numbers), where I give it an alphabet + length of the desired string. Using 8 to 12 characters, I can get a few million unique permutations. That is, if used as a primary key, few million per database table. Then I hear in the back of my head, guys from work who would argue I would run out of unique combinations or would have to do lookups to see if they exist. So I decided slap the year and month on it as a prefix, so a key might look like this: 2105HSUAMWPA. This gets indexed really well too and there is some inherent information that can be seen from looking at the key: Year 21, Month 5 and then the unique bits.It's basically 4 lines of code that gets called on every new database entity. I think it will be easy to shard/partition the data too if the need arise in the future, by simply looking at the first 4 digits.
Thus to summarize:
Data is sliced by entity type (customer, invoice, etc), then by date (2105 for May 2021) then by unique string.
What do you guys think about this approach? Anyone been burnt by something like this?
I don't think I've ever seen this mentioned anywhere, but if you need a unique ID for an entity with not a lot of records planned (≤10,000,000), why not use a random int64 with a simple for loop on the application side to catch the occasional collisions? Are there any downsides besides making the application side a tiny bit more complex?
Is the goal here to save space?
I once pondered how I might generate IDs that were as compact as a machine word, without a value (or small set of values) revealing the size of the data set. One application might be user-visible customer numbers that don't easily reveal how many customers there are. I eventually came across the idea of using maximal period linear-feedback shift registers to transform an integer variable through every possible value…
So far I haven't encountered any problems in the short term by using the approach described.
> Now, sometimes a table has a natural primary key, for example the social security number of a country’s citizens. You know, you think that, but it's never that simple. The field was added incorrectly and nobody noticed until the value is in countless tables that you now need to simultaneously update or the value is something that's supposed to be semi-secret, so now a low level support staff can't reference the row…
A projection in ES, is more a cache, not your primary store. The primary store is the eventlog. The latter should, obviuosly, never use natural ID's.
I have no particular expertise with modern databases and it has been decades since I did any work as a DBA. However, I cannot imagine creating table entries without a datestamp. No matter what else you are doing, or what you index by, I would want YYYY-MM-DD_HH-MM-SS in every row. Maybe I'm just weird that way ...
They are not serially incrementing but still sortable. Thus prevent index fragmentation issues observed with UUIDS. Are 8 bytes in length. So index size is smaller compared to UUIDs. So you get all benefits of serial IDs but they are not easily guessable thus preventing sequential access attacks.
Earlier quoted context omitted.
They almost got it right, a better implementation would overflow regularly to make use of the entire key space, and counter untuitively more resistant to overflows. Clocks aren't reliable enough for timestamps anyways so garbage collection is the only thing you kinda wanna rely on them for. A good sweet spot seems to be, 32bit milliseconds + 96bit of entropy. This overflows appeoximately every 50 days, allowing for 5…
Not the worst idea—50 days is a nice sweet spot between infrequent enough to have some indexing benefit and frequent enough that potential downsides will be discovered early in the product’s life cycle. Personally I wouldn’t do this. A scenario where for each individual millisecond of elapsed time, 96 bits of entropy is an upgrade over 80 bits of entropy, is fairly extreme. I don't think there are many databases in t…
Individual instances? Maybe not. But for those an autoincrement key would also work. That is not the scenario that ULIDs and GUIDs are advertised for.
The goal is to have an universally/globally unique ID. So whenever you encounter two IDs you can be (resonably, probability wise) sure that they won't collide.
Any such sheme thus must, by definition, serve every single use case now and forever everywhere. That's a tough one.
Also it's not really 80bit vs 96bit (which due to the birthday paradox is already a huge difference) but more 80bits vs. 128bit as the timestamp is recycled with sufficient usage.
I'm actually concerned that 96bit isn't enough, as it relies on the assumption that you'll use this scheme for for data spanning years, in order to properly use the timestamp as entropy.