Show HN: VectorVFS, your filesystem as a vector database
vectorvfs.readthedocs.io
Show HN: VectorVFS, your filesystem as a vector database
1–10 of 150 posts
Re: Show HN: VectorVFS, your filesystem as a vector database
#2Fun idea storing embeddings in inodes! Very clever!
I want to point out that this isn’t suitable for any kind of actual things you’d use a vector database for. There’s no notion of a search index. It’s always a O(N) linear search through all of your files: https://github.com/perone/vectorvfs/blob/main/vectorvfs/cli....
Still, fun idea :)
Re: Show HN: VectorVFS, your filesystem as a vector database
#3this is actually a great idea
Re: Show HN: VectorVFS, your filesystem as a vector database
#4Fun idea storing embeddings in inodes! Very clever! I want to point out that this isn’t suitable for any kind of actual things you’d use a vector database for. There’s no notion of a search index. It’s always a O(N) linear search through all of your files: https://github.com/perone/vectorvfs/blob/main/vectorvfs/cli.... Still, fun idea :)
Thanks. There is a bit of a nuance there, for example: you can build an index in first pass which will indeed be linear, but then later keep it in an open prompt for subsequent queries, I'm planning to implement that mode soon. But agree, it is not intended to search 10 million files, but you seldom have this use case in local use anyways.
Re: Show HN: VectorVFS, your filesystem as a vector database
#5Great idea indeed.
The documentation needs a bit more information to be useful.
What GPU backends are supported for example?
How do I delete the embedding information after I decide to uninstall it?
Will give it a try though.
Re: Show HN: VectorVFS, your filesystem as a vector database
#6Fun idea storing embeddings in inodes! Very clever! I want to point out that this isn’t suitable for any kind of actual things you’d use a vector database for. There’s no notion of a search index. It’s always a O(N) linear search through all of your files: https://github.com/perone/vectorvfs/blob/main/vectorvfs/cli.... Still, fun idea :)
O(n) is still OK for vector search if n isn't too large. Filesystem search solutions are currently terrible, with background indexing jobs and poor relevance. This won't scale for every file on your system but anything in your working documents folder would easily work well.
Re: Show HN: VectorVFS, your filesystem as a vector database
#7Great idea indeed. The documentation needs a bit more information to be useful. What GPU backends are supported for example? How do I delete the embedding information after I decide to uninstall it? Will give it a try though.
Thanks, I'm working on implementing the commands to clean the embeddings (you can now do that with Linux xattr command-line tool). I'm supporting CPU or GPU (NVIDIA) for the encoders and it only supports Linux at the moment.
Re: Show HN: VectorVFS, your filesystem as a vector database
#8Fun idea storing embeddings in inodes! Very clever! I want to point out that this isn’t suitable for any kind of actual things you’d use a vector database for. There’s no notion of a search index. It’s always a O(N) linear search through all of your files: https://github.com/perone/vectorvfs/blob/main/vectorvfs/cli.... Still, fun idea :)
The lack of an index is not bad at all if you have it stored contiguously in RAM: the mechanical sympathy is great, SIMD will spin like a top not to mention multithreaded programming, etc. Circa 2014 or so I worked on a search engine that scanned maybe 2GB worth of vectors for 10 million documents, queries were turned around in much less than a second, nobody complained about the speed.
If you gotta gather the data from a lot of different inodes, it is a different story.
Re: Show HN: VectorVFS, your filesystem as a vector database
#9interesting
Re: Show HN: VectorVFS, your filesystem as a vector database
#10The idea that filesystems are not just a flavor of database management systems was always a mistake.
Maybe with micro-kernels we'll finally fix this.