Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: VectorVFS, your filesystem as a vector database

vectorvfs.readthedocs.io

Post reply on HN
Print view View original

91–100 of 150 posts

Re: Show HN: VectorVFS, your filesystem as a vector database

#91

Post by quantadev »

I've been wondering for about 20 years why File Systems basically died and stopped innovating. For example we have lots of hierarchical data structures in the world, and no one seems to have figured out how to let a folder be the storage, instead of always just databases.

For example, if we simply had the ability to have "ordered" files inside folders, that would instantly make it practical for a folder structure to represent "Documents". After all, documents are nothing but a list of paragraphs and images, so if we simply had ordering in file systems we could have document editors which are using individual files for each paragraph of text or image. It would be amazing.

Also think about use cases like Jupyter Notebooks. We could stop using the XML file format, and just make it a folder structure instead. Each cell (node) being in a file. All social media messages and chatbot conversations could be easily saved as folders structures.

I've heard many file copy tools ignore XATTR so I've never tried to use it for this purpose, so maybe we've had the capability all along and just nobody thought to use it in a big way that became popular yet. Maybe I should consider XATTR and take it seriously.

Re: Show HN: VectorVFS, your filesystem as a vector database

#92

Post by Ericson2314 »

foobiekr wrote: post #79
jonhohle wrote: post #20

Earlier quoted context omitted.

BeOS got it right with BeFS. An Email client was just a folder. MP3s could be sorted and filtered in the file system. https://news.ycombinator.com/item?id=12309686

BeFS wasn't a database. It had indexed queries on EAs and they had the habit of asking application files to add their indexable content to the EAs. Internally it was just a mostly-not-transactional collection of btrees. There was no query language for updating files, or even inspecting anything about a file that was not published in the EAs (or implicitly do as with adapters), there were no multi-file transactions, n…

Yeah I am talking more deep architecture, and BeOS is more notable here mostly on just the user-interface level.

However, I think it is reasonable to think that with way more time and money, these things would meet up. Think about it as digging a tunnel from both sides of the mountain.

Re: Show HN: VectorVFS, your filesystem as a vector database

#93

Post by thirdtrigger »

Might be interesting to add an optional embedded Weaviate [1] with a flat-index [2] to the project. It wouldn't use external services and is fully disk-based. Would allow you to search the whole filesystem (about 1.5kb per file (384 dimensions) which would be added to the metadata as well).

1. https://weaviate.io/developers/weaviate/installation/embedde... 2. https://weaviate.io/developers/academy/py/vector_index/flat

Re: Show HN: VectorVFS, your filesystem as a vector database

#94

Post by lstodd »

mywittyname wrote: post #42
lstodd wrote: post #40

Earlier quoted context omitted.

so, like magic(5)?

What is magic(5) and how is it similar to what was described?

four people answered strictly correctly as to what magic(5) is, but not a single one realized that storing some aux data as xattr in linux FS is not in any way different from just storing the exact same data as a file header. which is how magic(5) works.

how come?

(besides good luck not forgetting to rsync those xattrs)

Re: Show HN: VectorVFS, your filesystem as a vector database

#95

Post by lstodd »

jlhawn wrote: post #38

If I understand correctly, this is attaching metadata to files in a format that LLMs (or any tool that can understand the semantic embedding vector) can leverage to understand what a file is without having to actually read the contents of the file. That obviously has a lot of interesting use cases, but my first assumption was that this could be used to quickly/easily search your filesystem with some prompt like "Play…

if you go look up how xattrs work, you will understand it's no different than just reading a chunk of the file in question, and in fact can be slower.

xattrs are better be forgotten already. it was just as dumb idea as macos resource forks/

Re: Show HN: VectorVFS, your filesystem as a vector database

#96

Post by binarymax »

thirdtrigger wrote: post #93

Might be interesting to add an optional embedded Weaviate [1] with a flat-index [2] to the project. It wouldn't use external services and is fully disk-based. Would allow you to search the whole filesystem (about 1.5kb per file (384 dimensions) which would be added to the metadata as well). 1. https://weaviate.io/developers/weaviate/installation/embedde... 2. https://weaviate.io/developers/academy/py/vector_index/fla…

Why weaviate and not FAISS? The latter is faster and lighter.

Re: Show HN: VectorVFS, your filesystem as a vector database

#99

Post by jlhawn »

perone wrote: post #56
b0a04gl wrote: post #14

If VectorVFS obscures retrieval logic behind opaque embeddings, how do users debug why a file surfaced—or worse, why one didn’t?

Hi, not sure if I understood what you meant by opaque embeddings as well, but the reason why files surface or not is due to the similarity score (which is basically the dot product of embeddings).

How much work do you think it would be to also have a separate xattr which has a human-readable description of the file contents? I wonder if it that might already be an intermediate product of some of the embedding tools, like "arbitrary media" -> "text description of media" -> "embedding vector". You could store both of those as xattrs and you could debug by comparing your text query with the text description of the file contents as they should produce similar embedding vectors. You could even audit any file, assuming you know what its contents are, by checking the text description xattr generated by this program.

Re: Show HN: VectorVFS, your filesystem as a vector database

#100

Post by j45 »

gitroom wrote: post #97

Gotta say, the old school debate on filesystems vs databases will never get old for me - I always end up with more questions than answers after reading stuff like this.

Everything's old school, everything's new.

It's important to remember that the cloud is also invented by the old school and understanding the oscillation between client/server architectures vs local, and it's implication on topics of data and files is interesting too.

More questions means more learning until I learned there's no one right or wrong, just what works best, where, when, for how long, and what the tradeoffs are.

Quick wins/decisions are often bandaids that pile up in an different way.

Post reply on HN