Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There's no single best way to store information

quantamagazine.org

Post reply on HN
Print view View original

41–50 of 53 posts

Re: There's no single best way to store information

#41

Post by altmanaltman »

akhil08agrawal wrote: post #22

This clicked for me in a way I didn't expect. I've been thinking about trade-offs as "pick two of three" in the abstract, but the bookshelf example made it concrete. The insight that matters is: if you know your query patterns, you can optimize differently. As a PM, I keep trying to build systems that work for "every case." But this article reminded me that's the wrong goal. The hash table works because it accepts th…

You're a PM and this basic-level watered down article barely discussing anything "clicked for you in a way" you didn't expect? Of course the best system is desinged based on requirements, how can a PM not know this before being a PM?

Re: There's no single best way to store information

#42

Post by 1970-01-01 »

megaBiteToEat wrote: post #40
1970-01-01 wrote: post #33

Earlier quoted context omitted.

It's not pedantic, you are correctly using words as we understand them, and they are not. The headline needs a sharp correction. Editing jobs are in very short supply these days.

Oh come on. Programmers discuss how to "store" data in memory as a data model all the time. You're reducing definitions and meaning too far to make an ultimately empty point just to contribute the thread. If social medias only contribution is language policing, then it really should die off. What a waste of resources so functional illiterate nobodies can project ego.

No, I'll think I'll double down, because I do think I'm right here.

https://en.wikipedia.org/wiki/Data_storage is a different website from https://en.wikipedia.org/wiki/Data_store because they are different, slightly overlapping concepts.

Re: There's no single best way to store information

#43

Post by teraflop »

__MatrixMan__ wrote: post #3

There are, however, several objectively bad ways. In "Service Model" (a novel that I recommend) a certain collection of fools decides to sort bits by whether it's a 1 or a 0, ending up with a long list of 0's followed by a long list of 1's.

There's a similar anecdote in Iain M. Banks' The Player of Games.

https://scifi.stackexchange.com/questions/270578/negotiator-...

Re: There's no single best way to store information

#44

Post by vee-kay »

Millions of years of evolution has resulted in the human brain being the best way to store information.

I doubt we humans will be able to do better (faster, more capacity, more analytical, more intuitive, more logical) storage (at an individual level, not at mass scale, since that's kinda achieved already by the behemoths like Google, etc.) in a few thousand years of civilization.

Quantum computing may be the game changer though.

I read somewhere that the entirety of humanity's information, including all knowledge and data of past (of every human ever) and current, if stored via quantum computing - that quanta of quantum information will just be the size of a football.

Re: There's no single best way to store information

#45

Post by sandworm101 »

DixieDev wrote: post #21
bob1029 wrote: post #11

The best way to store information depends on how you intend to use (query) it. The query itself represents information. If you can anticipate 100% of the ways in which you intend to query the information (no surprises), I'd argue there might be an ideal way to store it.

This line of thought works for storage in isolation, but does not hold up if write speed is a concern.

Speed can always be improved. If a method is too slow, run multiple machines in parralel. Longevity is different as it cannot scale. A million cd burners are together very fast, but the CDs wont last any longer. So the storage method is is the more profound tech problem.

Re: There's no single best way to store information

#46

Post by roenxi »

bob1029 wrote: post #11

The best way to store information depends on how you intend to use (query) it. The query itself represents information. If you can anticipate 100% of the ways in which you intend to query the information (no surprises), I'd argue there might be an ideal way to store it.

Yes with the important caveat that a lot of the time people don't have a crystal ball, can't see the far future, don't know if their intents will materialise in practice 12 months down the line and should therefore store information in Postures until that isn't a feasible option any more.

A consequence of there being no generally superior storage mechanism is that technologists as a community should have an agreed default standard for storage - which happens to be relational.

Re: There's no single best way to store information

#47

Post by jandrewrogers »

bob1029 wrote: post #11

The best way to store information depends on how you intend to use (query) it. The query itself represents information. If you can anticipate 100% of the ways in which you intend to query the information (no surprises), I'd argue there might be an ideal way to store it.

This is connected to the equivalence relationship between optimal indexing and optimal AGI. The "best" way is optimal for the entire universe of possible queries but has the downside of being profoundly computationally intractable.

Requiring perfect knowledge of how information will be used is brittle. It has the major benefit of making the algorithm design problem tractable, which is why we do it.

An alternative approach is to exclude large subsets of queries from the universe of answerable queries without enumerating the queries that the system can answer. The goal is to qualitatively reduce the computational intractability of the universal case by pruning it without over-specifying the queries it can answer such as in the traditional indexing case. This is approximately what "learned indexing" attempts to do.

Re: There's no single best way to store information

#48

Post by ssivark »

bob1029 wrote: post #11

The best way to store information depends on how you intend to use (query) it. The query itself represents information. If you can anticipate 100% of the ways in which you intend to query the information (no surprises), I'd argue there might be an ideal way to store it.

What if the various potential queries demand different / conflicting compression schemes?

I'd say this is spiritually what the no-free-lunch theorems are about... Because whatever "AI model" / query system you build -- it is implicitly biased towards queries coming from one slice of futures.

Re: There's no single best way to store information

#49

Post by ifh-hn »

nicbou wrote: post #19
ronsor wrote: post #16

There are plenty of good enough ways: * For lossless compression of generic data, gzip or zstd. * For text, documentation, and information without fancy formatting, markdown, which is effectively a plain-text superset. * For small datasets, blobs, objects, and what not, JSON. * For larger datasets and durable storage, SQLite3. Whenever there's text involved, use UTF-8. Whenever there's dates, use ISO8601 format (UTC…

One format I'm missing: storage for conversations and social media posts. Both are complex media (text + images/videos + metadata), and one is actually a collection of such posts. How would you go about storing those in a somewhat human-readable format? My goal is to archive my chats and social media activity.

Depends on what you mean by humans readable. Sqlite, as the other sub comment mentions is good, but you could also just use a CSV file, unnormalised table, and sit the original media in the same or a sub folder. Hell, convert that CSV to a html table and you can display the data as a human readable local webpage. Through in some JS and you can navigate/filter it too.

Re: There's no single best way to store information

#50

Post by __MatrixMan__ »

HPsquared wrote: post #23
__MatrixMan__ wrote: post #3

There are, however, several objectively bad ways. In "Service Model" (a novel that I recommend) a certain collection of fools decides to sort bits by whether it's a 1 or a 0, ending up with a long list of 0's followed by a long list of 1's.

That's fine so long as there's an index!

Presumably there was at some time, but they put it in long term bit storage also.
Post reply on HN