If there is no indexing, how will your search time not increase linear or worse with the number of files?
Show HN: VectorVFS, your filesystem as a vector database
121–130 of 150 posts
Re: Show HN: VectorVFS, your filesystem as a vector database
#122Might be interesting to add an optional embedded Weaviate [1] with a flat-index [2] to the project. It wouldn't use external services and is fully disk-based. Would allow you to search the whole filesystem (about 1.5kb per file (384 dimensions) which would be added to the metadata as well). 1. https://weaviate.io/developers/weaviate/installation/embedde... 2. https://weaviate.io/developers/academy/py/vector_index/fla…
Why weaviate and not FAISS? The latter is faster and lighter.
It depends on additional filters and whether you want to use vector search only. The upside of using Faiss would be storing the ID as file metadata and embedding it in the Faiss index. However, if you need any other filters or data, you would need to store it somewhere else.
Re: Show HN: VectorVFS, your filesystem as a vector database
#123I've been wondering for about 20 years why File Systems basically died and stopped innovating. For example we have lots of hierarchical data structures in the world, and no one seems to have figured out how to let a folder be the storage, instead of always just databases. For example, if we simply had the ability to have "ordered" files inside folders, that would instantly make it practical for a folder structure to…
I agree! I'm sort of exploring "programmable filesystem" concept (using FuseFS) (for some notes, see [1]).
Re: ordered files: depends on FS. e.g. filesystems which use B+ trees will tend to have files (in directories) in lexical order. So in some cases you may not need a new FS:
echo 'for f in *.txt; do cat "$f"; done' > doc.sh; chmod +x doc.sh
That said... wouldn't it be nice to have ridiculous easily pluggable features like
echo "finish this poem: roses are red," > /auto-llm/poem.txt; cat ..
[1]: chaotic notes: https://kfs.mkj.lt/#welcome (see bullet point list below)
Re: Show HN: VectorVFS, your filesystem as a vector database
#124I've long wanted to have a linux filesystem that robustly supported "tags" for files so that I didn't have to rely on the filesystem hierarchy to represent media files etc. e.g. I might want to tag a particular films as "Scifi" and also "Horror". Of course, for films, NFO files are typically used for this kind of metadata, but I'd like a similar facility that could be applied to any type of file.
Re: Show HN: VectorVFS, your filesystem as a vector database
#125Earlier quoted context omitted.
Hi, it is quite different, there is no LLM involved, we can certainly use it for a RAG for example, but what is currently implemented is basically a way to generate embeddings (vector representation) which are then used for search later, it is all offline and local (no data is ever sent to cloud from your files).
I understand that LLMs aren't involved in generating the embeddings and adding the xattrs. I was just wondering what the value add of this is if there's no other background process (like mds on macOS) which is using it to build a search index. I guess what I'm asking is: how does VectorVFS enable search besides iterating through all files and iteratively comparing file embeddings with the embedding of a search query?…
Yeah this kind of setup is indefinitely scaleable, but not searchable without out a meta db/index keeping track of all the nodes.
Re: Show HN: VectorVFS, your filesystem as a vector database
#126I've long wanted to have a linux filesystem that robustly supported "tags" for files so that I didn't have to rely on the filesystem hierarchy to represent media files etc. e.g. I might want to tag a particular films as "Scifi" and also "Horror". Of course, for films, NFO files are typically used for this kind of metadata, but I'd like a similar facility that could be applied to any type of file.
That is literally what xattrs are for.
Re: Show HN: VectorVFS, your filesystem as a vector database
#127Earlier quoted context omitted.
It was abandoned due to The Cloud. There was no need for WinFS as a tech when you could store everything in The Cloud. It was also complex, ran poorly, and would have required developers to integrate their applications. Microsoft had long solved the problem of blobs and metadata in ESE and SharePoint's use of MS SQL for binary + metadata storage.
> it was just a SQL database that stored arbitrary data. I mean, for some definitions of “just”, “SQL database”, and “arbitrary data.” :) It was a schematised graph database implemented on top of a slimmed-down version of SQL Server. The query language was not SQL-based. > It was abandoned due to The Cloud. It was discontinued circa 2007. The cloud was much less of a Thing back then. I don’t recall that factoring at…
Did Microsoft tell the WinFS team why they decided to cancel the product? If so, can you reveal what the real reason was?
Re: Show HN: VectorVFS, your filesystem as a vector database
#128Earlier quoted context omitted.
> it was just a SQL database that stored arbitrary data. I mean, for some definitions of “just”, “SQL database”, and “arbitrary data.” :) It was a schematised graph database implemented on top of a slimmed-down version of SQL Server. The query language was not SQL-based. > It was abandoned due to The Cloud. It was discontinued circa 2007. The cloud was much less of a Thing back then. I don’t recall that factoring at…
Did Microsoft tell the WinFS team why they decided to cancel the product? If so, can you reveal what the real reason was?
I mean, it’s not like whenever a project is cancelled at Microsoft they bring team members into a room and say, “We’re totally going to lie to the public (and therefore the shareholders) but here’s the real reason...”
Maybe all the nuances aren't fully communicated publicly when a project is cancelled, but I don’t recall having a sense that what was said publicly was any different than our understanding internally. But that was almost 20 years ago.
The majority of the teams I was on during my time there were ‘internal startups’: Mira, NetGen, WinFS, MatrixDB. Like startups anywhere, projects being unceremoniously cancelled was par for the course.
Re: Show HN: VectorVFS, your filesystem as a vector database
#129I've long wanted to have a linux filesystem that robustly supported "tags" for files so that I didn't have to rely on the filesystem hierarchy to represent media files etc. e.g. I might want to tag a particular films as "Scifi" and also "Horror". Of course, for films, NFO files are typically used for this kind of metadata, but I'd like a similar facility that could be applied to any type of file.
That is literally what xattrs are for.
Yes, but they seem fairly limited in terms of userspace programs. How would you use xattrs to produce a filesystem hierarchy that say, listed the same file in multiple folders according to the attributes?
Re: Show HN: VectorVFS, your filesystem as a vector database
#130Earlier quoted context omitted.
That doesn't disprove anything for me. It just says POSIX DOS lowest common denominator network effects are a hell of a drug. Whenever we're talking about interfaces, coordination success or failure is the name of the game.
What problem do you think a DB-as-filesystem solves? The only obvious one that makes any sense at all is cross-file transactions.
The filesystem is a bad database
Directories are a shitty underpowered way to organize data?
No good transactions
Conflation of different needs such as atomic replace vs log-structured
I would like to use a better database instead.