21–30 of 53 posts
The best way to store information depends on how you intend to use (query) it. The query itself represents information. If you can anticipate 100% of the ways in which you intend to query the information (no surprises), I'd argue there might be an ideal way to store it.
I've been thinking about trade-offs as "pick two of three" in the abstract, but the bookshelf example made it concrete. The insight that matters is: if you know your query patterns, you can optimize differently.
As a PM, I keep trying to build systems that work for "every case." But this article reminded me that's the wrong goal. The hash table works because it accepts the space-time trade-off. The heap works because it embraces disorder for non-priority items.
Sometimes the best system isn't the most elegant one—it's the one that matches how you'll actually use it.
Good reminder to stop over-optimizing for flexibility I'll never need.
Thanks for sharing.
There are, however, several objectively bad ways. In "Service Model" (a novel that I recommend) a certain collection of fools decides to sort bits by whether it's a 1 or a 0, ending up with a long list of 0's followed by a long list of 1's.
There are, however, several objectively bad ways. In "Service Model" (a novel that I recommend) a certain collection of fools decides to sort bits by whether it's a 1 or a 0, ending up with a long list of 0's followed by a long list of 1's.
It _does_ open up amazing opportunities for compression though.
Conceptually similar to CAP, but with storage trade-offs. The idea is you can only pick 2 out of 3.
The best way to store information depends on how you intend to use (query) it. The query itself represents information. If you can anticipate 100% of the ways in which you intend to query the information (no surprises), I'd argue there might be an ideal way to store it.
This line of thought works for storage in isolation, but does not hold up if write speed is a concern.
I'd love to be clued in on more interesting architectures that either attempt to optimize both or provide a more continuous tuning knob between them
There are plenty of good enough ways: * For lossless compression of generic data, gzip or zstd. * For text, documentation, and information without fancy formatting, markdown, which is effectively a plain-text superset. * For small datasets, blobs, objects, and what not, JSON. * For larger datasets and durable storage, SQLite3. Whenever there's text involved, use UTF-8. Whenever there's dates, use ISO8601 format (UTC…
One format I'm missing: storage for conversations and social media posts. Both are complex media (text + images/videos + metadata), and one is actually a collection of such posts. How would you go about storing those in a somewhat human-readable format? My goal is to archive my chats and social media activity.
There are plenty of good enough ways: * For lossless compression of generic data, gzip or zstd. * For text, documentation, and information without fancy formatting, markdown, which is effectively a plain-text superset. * For small datasets, blobs, objects, and what not, JSON. * For larger datasets and durable storage, SQLite3. Whenever there's text involved, use UTF-8. Whenever there's dates, use ISO8601 format (UTC…
One format I'm missing: storage for conversations and social media posts. Both are complex media (text + images/videos + metadata), and one is actually a collection of such posts. How would you go about storing those in a somewhat human-readable format? My goal is to archive my chats and social media activity.
You can have another table for attachments (images, videos, etc.). If they're small, store them directly in a BLOB. If they're not, store them alongside the database, and only store the relative path in the attachments table.
You may opt to convert images and videos to a single format (e.g. PNG and H.264 MP4), but you can lose information depending on the target format. It may be preferable to leave them in the original (or highest quality) format.
Earlier quoted context omitted.
One format I'm missing: storage for conversations and social media posts. Both are complex media (text + images/videos + metadata), and one is actually a collection of such posts. How would you go about storing those in a somewhat human-readable format? My goal is to archive my chats and social media activity.
Why not just use WARC and a program that can read them? Do archives need to be human-readable?
Or it's the opposite, where the slowest possible retrieval time is the intended effect, as is the basis of many cryptographic algorithms.
(Or I guess, more generally, the intended effect is zero correlation between the information and the time it takes to retrieve it. If retrieval time were completely random, it would achieve the goal, but it wouldn't have zero variation.)