261–270 of 376 posts
Earlier quoted context omitted.
Yes but if that machine with sequential data receives 100x the traffic of other machines, it can be worse than splitting this traffic evenly across all available machines.
It depends if you have a request covers a lot of sequential data, or if you have a lot of requests of sequential data.
Earlier quoted context omitted.
Yes but if that machine with sequential data receives 100x the traffic of other machines, it can be worse than splitting this traffic evenly across all available machines.
If your database simply shards keys sequentially, it's going to get hotspots in a lot of use cases, like plain old integer keys and timestamps, not just UUIDv7. In that case it would be fair to say that your database is doing it wrong. Fortunately, there's no rule that says you should shard your keys using the sequential part up front. One of the rules for generating randomness from environmental sources is to throw…
Earlier quoted context omitted.
Having sequential ID's is more than just a security risk, it's an information risk. Competitors can use them to estimate the size of your business, the number of customers you have, and all sorts of stuff. This was used in the war to estimate the number of German tanks based on the sequential IDs https://en.wikipedia.org/wiki/German_tank_problem So just for business intelligence you don't want to leak your IDs.
I’ve heard this argument many times, but I’ve never seen anyone actually post a reference to it happening (as in, a company finding and using this information; not the German tank problem). To me, it reeks of solving imaginary problems while causing new ones.
Earlier quoted context omitted.
Yep have used an approach just like that, worked quite well if you have a strong pattern to easily translate from one to the other. Gives you an id with the right properties for internal use, efficient indexing etc, and in its encrypted form gives you the properties you want from an external identifier being unpredictable etc, all from one source id. It is true that now your encryption key is now very long lived and…
>> It is true that now your encryption key is now very long lived and effectively part of your public interface No need to encrypt, just store the external key in a table. Not that you're likely to change algorithms.
Earlier quoted context omitted.
jonhohle, thanks. Do you know of examples of when milliseconds are part of the session tokens or accounts being created has been exploited?
I know of people who used leaked customer ids in public facing chatbot solutions (like Intercom) to estimate how fast their competitors were growing and/or how many customers they had.
Is there some reason new versions of UUID keep appearing? It seems like the desired properties are never quite achieved so new ones appear later. Is there a table with UUID version across the top and characteristics down the side, so I can see the differences and pick one that fits my needs? That might also help to explain why there are so many variants.
v1: mac address + time + random
v4: completely random
v5: input + seed (consistent, derived from input)
v7: time + random (distributed sortable ids)
Earlier quoted context omitted.
Reading their docs: No real benefits, just misconceptions. 1. Collision resistance / "weak" PRGNs used to generate UUIDv4. Firstly, these are properties of the implementation , not the spec. Secondly, the source for calling the browser `Crypto.getRandomValues()` insecure is an issue that has been fixed back in 2016. I would not trust the developers of this implementation to do a better job than current browsers. 2. "…
No argument given for why having a slower algorithm to generate random ids is more secure. If the algorithm is too fast it means you can detect when some other part of the system is having a significant impact on how the key is returned. Eg checking a database to see if a user exists and returning their key versus getting null back and generating a new key. That difference can be used to determine if a user exists. Y…
ID generation should usually only happen when creating new assets, so it should be as fast as possible.
Earlier quoted context omitted.
> What's the benefit of the internal structure at all? Purely random identifiers are the bane of DB indices. The internal structure is sequential-ish and therefore indexes well.
Purely random identifiers are the recommended primary key for plenty of databases - eg. spanner. Random identifiers spread work evenly between shards of something sharded by keyspace. They also don't get 'hotspotting' on more recent records - recent records frequently get more than their fair share of updates and changes, and database query planners have no knowledge of that.
For large distributed systems like Spanner you want to avoid a hotspot on a single node as that limits throughout.
However for a single-node system like PostgreSQL you want hot spots because they are cache friendly. (Locks aside)
Basically you want a hotspot that is as hot as a single node can manage, but no hotter. Even for things like Spanner you still want good cache hits (which is why they support table interleaving and other methods of clustering related data) but in general avoiding overloading single nodes is more important (as it hurts latency and efficiency but doesn't limit throughput like a hotspot would).
Earlier quoted context omitted.
Purely random identifiers are the recommended primary key for plenty of databases - eg. spanner. Random identifiers spread work evenly between shards of something sharded by keyspace. They also don't get 'hotspotting' on more recent records - recent records frequently get more than their fair share of updates and changes, and database query planners have no knowledge of that.
Spanner is also bespoke, and was probably designed with that in mind. Anything with a clustering index - MySQL (with InnoDB), SQL Server - will absolutely hate the page splits from a random PK. Postgres also doesn’t like it that much, for a variety of reasons. Tuples are in a heap, but the PK is still a B+tree (ish), so it still suffers from splits. They also use half the default page size as MySQL, AND their MVCC im…
For tables this is not such a problem because of reasonably good locality (rows inserted close in time will end up close in the file too), but for indexes that's very difference. In Postgres this is particularly painful for writes, because of the write amplification.
None of this really breaks sharding (IDs are still unique and can be hashed), so no new hotspots. It can't fix the "many rows with the same ID" issue, but neither can any other ID.
Earlier quoted context omitted.
Given that a UUID identifier fits in a single cipher block, and the whole point is that these are unique by construction (no IV needed so long as that holds true), it seems like a single round of ECB-mode AES-128 would enable quickly converting between internal/external identifiers. 128 bits -> 128 bits
Why not just use the AES-128 result as the UUID then? What's the benefit of the internal structure at all? If AES-128 is an acceptable external UUID (and likely an acceptable internal one), then you might as well just stick with a faster RNG.