Show HN: VectorVFS, your filesystem as a vector database • Hacker News Board

Re: Show HN: VectorVFS, your filesystem as a vector database

#61

Post by Ericson2314 » Mon, May 05, 2025, 5:38 PM UTC

adolph wrote: post #21

foobiekr wrote: post #16

Earlier quoted context omitted.

Every single time this has been tried it has gone wrong, but sure. Almost all of the operations done on actual filesystems are not database like, they are close to the underlying hardware for practical reasons. If you want a database view, add one in an upper layer.

> they are close to the underlying hardware for practical reasons Could you provide reference information to support this background assertion? I'm not totally familiar with filesystems under the hood, but at this point doesn't storage hardware maintain an electrical representation relatively independent from the logical given things like wear leveling?

Yes I agree, that assertion doesn't pass muster.

Mature database implementations also bypass a lot of kernel machinary to get closer to the underlying block devices. The layering of DB on top of FS is a failure.

Re: Show HN: VectorVFS, your filesystem as a vector database

#62

Post by sunshine-o » Mon, May 05, 2025, 5:38 PM UTC

tzury wrote: post #27

I’ve found that starting with a plain old filesystem often outperforms fancy services - just as the Unix philosophy (“everything is a file” [1]) has preached for decades [2]. When BigQuery was still in alpha I had to ingest ~15 billion HTTP requests a day (headers, bodies, and metadata). None of the official tooling was ready, so I wrote a tiny bash script that: 1. uploaded the raw logs to Cloud Storage, and 2. track…

Absolutely.

I would add that filesystems are superior to data formats (XML, JSON, YAML, TOML) for many use cases such as configuration or just storing data.

- Hierarchy are dirs,

- Keys are file names,

- Value is the content of the file.

- Other metadata are in hidden files

It will work forever, you can leverage ZFS, Git, rsync, syncthing much better. If you want, a fancy shells like Nushell will bring the experience pretty close to a database.

Most important you don't need fancy editor plugins or to learn XPath, jq or yq.

Re: Show HN: VectorVFS, your filesystem as a vector database

#63

Post by jlhawn » Mon, May 05, 2025, 5:39 PM UTC

perone wrote: post #49

jlhawn wrote: post #38

If I understand correctly, this is attaching metadata to files in a format that LLMs (or any tool that can understand the semantic embedding vector) can leverage to understand what a file is without having to actually read the contents of the file. That obviously has a lot of interesting use cases, but my first assumption was that this could be used to quickly/easily search your filesystem with some prompt like "Play…

Hi, it is quite different, there is no LLM involved, we can certainly use it for a RAG for example, but what is currently implemented is basically a way to generate embeddings (vector representation) which are then used for search later, it is all offline and local (no data is ever sent to cloud from your files).

I understand that LLMs aren't involved in generating the embeddings and adding the xattrs. I was just wondering what the value add of this is if there's no other background process (like mds on macOS) which is using it to build a search index.

I guess what I'm asking is: how does VectorVFS enable search besides iterating through all files and iteratively comparing file embeddings with the embedding of a search query? The project description says "efficient and semantically searchable" and "eliminating the need for external index files or services" but I can't think of any more efficient way to do a search without literally walking the entire filesystem tree to look for the file with the most similar vector.

Edit: reading the docs [1] confirmed this. The `vfs search TERM DIRECTORY` command:

> will automatically iterate over all files in the folder, look for supported files and then embed the file or load existing embeddings directly from the filesystem."

[1]: https://vectorvfs.readthedocs.io/en/latest/usage.html#vfs-se...

Re: Show HN: VectorVFS, your filesystem as a vector database

#64

Post by perone » Mon, May 05, 2025, 5:39 PM UTC

iugtmkbdfil834 wrote: post #15

natas wrote: post #3

this is actually a great idea

Assuming I understand it correctly, the idea is to be able to have LLMs get through file systems more easily with some interesting benefits to human users as well. The idea is interesting and I want to try it out.

Hi, there are no LLMs involved, it is all local and an embedding (vector representation) of the data is created and then that is used for search later, nothing is sent to cloud from your files and there are no local LLMs running as well, only the encoders (I use the Perception Encoder from Meta released a few weeks ago).

Re: Show HN: VectorVFS, your filesystem as a vector database

#65

Post by perone » Mon, May 05, 2025, 5:39 PM UTC

adenta wrote: post #25

I wonder if I could use this locally on my macbook. The finder applications built-in search is kinda meh.

I'm planning to support MacOS, the only issue is with the encoders that I'm using now, I will probably work more on it next week to try to make a release that works on MacOS as well. Thanks !

Re: Show HN: VectorVFS, your filesystem as a vector database

#66

Post by packetlost » Mon, May 05, 2025, 5:40 PM UTC

foobiekr wrote: post #16

Ericson2314 wrote: post #10

The idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.

Every single time this has been tried it has gone wrong, but sure. Almost all of the operations done on actual filesystems are not database like, they are close to the underlying hardware for practical reasons. If you want a database view, add one in an upper layer.

I don't see how file systems aren't some sort of DBMS, definitely not relational but that wasn't a stated requirement.

Re: Show HN: VectorVFS, your filesystem as a vector database

#67

Post by perone » Mon, May 05, 2025, 5:41 PM UTC

pseudosavant wrote: post #52

This immediately made me nostalgic for BeOS's BeFS or Windows Longhorn's WinFS database filesystems, and how this kind of thing would have fit them perfect. So much cool stuff you could do with vectors for everything. Smart folders that include files for a project based on a description of the project. Show me all of my config files for appXYZ. Images of a black dog at the beach. At the OS-level for any other app to…

I share the same feeling, I think filesystems will have to reinvent themselves given the pace of how useful ML models became in the past years.

Re: Show HN: VectorVFS, your filesystem as a vector database

#68

Post by mrlongroots » Mon, May 05, 2025, 5:42 PM UTC

adolph wrote: post #21

foobiekr wrote: post #16

Earlier quoted context omitted.

Every single time this has been tried it has gone wrong, but sure. Almost all of the operations done on actual filesystems are not database like, they are close to the underlying hardware for practical reasons. If you want a database view, add one in an upper layer.

> they are close to the underlying hardware for practical reasons Could you provide reference information to support this background assertion? I'm not totally familiar with filesystems under the hood, but at this point doesn't storage hardware maintain an electrical representation relatively independent from the logical given things like wear leveling?

Some examples off the top of my head:

- You can reason about block offsets. If your writes are 512B-aligned, you can be ensured minimal write amplification.

- If your writes are append-only, log-structured, that makes SSD compaction a lot more straightforward

- No caching guarantees by default. Again, even SSDs cache writes. Block writes are not atomic even with SSDs. The only way to guarantee atomicity is via write-ahead logs.

- The NVMe layer exposes async submission/completion queues, to control the io_depth the device is subjected to, which is essential to get max perf from modern NVMe SSDs. Although you need to use the right interface to leverage it (libaio/io_uring/SPDK).

Re: Show HN: VectorVFS, your filesystem as a vector database

#69

Post by p_ing » Mon, May 05, 2025, 5:47 PM UTC

pseudosavant wrote: post #52

This immediately made me nostalgic for BeOS's BeFS or Windows Longhorn's WinFS database filesystems, and how this kind of thing would have fit them perfect. So much cool stuff you could do with vectors for everything. Smart folders that include files for a project based on a description of the project. Show me all of my config files for appXYZ. Images of a black dog at the beach. At the OS-level for any other app to…

WinFS wasn't a file system laid down on hardware, it was just a SQL database that stored arbitrary data.

Re: Show HN: VectorVFS, your filesystem as a vector database

#70

Post by esafak » Mon, May 05, 2025, 5:50 PM UTC

perone wrote: post #60

esafak wrote: post #19

Files-as-vector stores is LanceDB's value proposition. How do you compare in performance, etc.?

This is quite different than LanceDB. In VectorVFS I'm using the inodes directly to store the embeddings, there is no external file with metadata and db, the db is your filesystem itself, that's the key difference.

That's an implementation detail, and it sounds more like a liability than a selling point, to have such tight coupling. (Why) do you see not using files as a good thing?

Let me ask another question: is this intended for production use, or is it more of a research project? Because as a user I care about things like speed, simplicity, flexibility, and robustness.