That is fair, I HAVE INDEED seen similar of those anecdotes. However, I still am skeptical of the file analogy.
LLMs are good at traversing traditional file systems because file/tree-like structures very likely encompass an overwhelming supermajority of their training set. By contrast, other approaches like graph-based vector/data discovery via sql queries seem equally or more promising (to me), not to mention the ability to run curl/http queries. Either way, it’s still all prompt engineering approaches to discover/manage context. In this way, files seem like a “local minimum” ie a medium that dually optimizes interpretability and accessibility for both LLMs AND humans.
Just riffing here, but this could even broadly be considered a discussion of memory-vs-storage (somewhat analogous to fluid vs crystallized memory in humans). In this way, one could imagine the models performing context compaction/backup by periodically dumping their context to files (or some other non-volatile storage) WITHOUT coming back to feature space.. just dump/load the tokens directly.
Personally, I am much more interested in even other approaches like VLMs (using visual tokens), architectures like auto encoders and its variations, jepa architectures and other approaches that emphasize operating primarily within the latent space.
To be fair, my particular interests have always been more in the computer vision area, and have been amused to watch the attention mechanism rise to prominence even over CNNs (given the contrast |similarity in their mechanics). Then again, I began my journey in the ML field when GANs were still the hotness, but I digress……
for LLMs, it’s tokens all the way down, and the name of the game is how discoverable and accessible can you make them?