Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: VectorVFS, your filesystem as a vector database

vectorvfs.readthedocs.io

Post reply on HN
Print view View original

81–90 of 150 posts

Re: Show HN: VectorVFS, your filesystem as a vector database

#81

Post by foobiekr »

Ericson2314 wrote: post #61
adolph wrote: post #21

Earlier quoted context omitted.

> they are close to the underlying hardware for practical reasons Could you provide reference information to support this background assertion? I'm not totally familiar with filesystems under the hood, but at this point doesn't storage hardware maintain an electrical representation relatively independent from the logical given things like wear leveling?

Yes I agree, that assertion doesn't pass muster. Mature database implementations also bypass a lot of kernel machinary to get closer to the underlying block devices. The layering of DB on top of FS is a failure.

You are confusing that databases implement their own filesystem equivalent functionality in an application-specific way with the idea that FS's can or should be databases.

Re: Show HN: VectorVFS, your filesystem as a vector database

#83

Post by [deleted] »

jlhawn wrote: post #63
perone wrote: post #49

Earlier quoted context omitted.

Hi, it is quite different, there is no LLM involved, we can certainly use it for a RAG for example, but what is currently implemented is basically a way to generate embeddings (vector representation) which are then used for search later, it is all offline and local (no data is ever sent to cloud from your files).

I understand that LLMs aren't involved in generating the embeddings and adding the xattrs. I was just wondering what the value add of this is if there's no other background process (like mds on macOS) which is using it to build a search index. I guess what I'm asking is: how does VectorVFS enable search besides iterating through all files and iteratively comparing file embeddings with the embedding of a search query?…

[deleted]

Re: Show HN: VectorVFS, your filesystem as a vector database

#84

Post by cryptonector »

sunshine-o wrote: post #62
tzury wrote: post #27

I’ve found that starting with a plain old filesystem often outperforms fancy services - just as the Unix philosophy (“everything is a file” [1]) has preached for decades [2]. When BigQuery was still in alpha I had to ingest ~15 billion HTTP requests a day (headers, bodies, and metadata). None of the official tooling was ready, so I wrote a tiny bash script that: 1. uploaded the raw logs to Cloud Storage, and 2. track…

Absolutely. I would add that filesystems are superior to data formats (XML, JSON, YAML, TOML) for many use cases such as configuration or just storing data. - Hierarchy are dirs, - Keys are file names, - Value is the content of the file. - Other metadata are in hidden files It will work forever, you can leverage ZFS, Git, rsync, syncthing much better. If you want, a fancy shells like Nushell will bring the experience…

Except that now when you do need a tool like XSLT/XPath, jq, or yq, now you need bash. I use bash lots, but still I'd rather use a better language, like the ones you listed.

I'm being slightly hypocritical because I've made plenty of use of the filesystem as a configuration store. In code it's quite easy to stat one path relative to a directory, or open it and read it, so it's very tempting.

Re: Show HN: VectorVFS, your filesystem as a vector database

#86

Post by pilooch »

perone wrote: post #49
jlhawn wrote: post #38

If I understand correctly, this is attaching metadata to files in a format that LLMs (or any tool that can understand the semantic embedding vector) can leverage to understand what a file is without having to actually read the contents of the file. That obviously has a lot of interesting use cases, but my first assumption was that this could be used to quickly/easily search your filesystem with some prompt like "Play…

Hi, it is quite different, there is no LLM involved, we can certainly use it for a RAG for example, but what is currently implemented is basically a way to generate embeddings (vector representation) which are then used for search later, it is all offline and local (no data is ever sent to cloud from your files).

Using it for a RAG is smart indeed, especially with a multimodal encoder (vision-rag), as the implementation would be straightforward from what you already have.

Re: Show HN: VectorVFS, your filesystem as a vector database

#87

Post by didgetmaster »

tugdual wrote: post #76
didgetmaster wrote: post #74

Earlier quoted context omitted.

I built a local object store that was designed to replace file systems. You can create hundreds of millions of objects (e.g. files) and attach a variety of metadata tags to each one. A tag could be a number, string, or other data type (including vector info). Searches for objects with certain tags is exceptionally fast. I invented it because I found searching conventional file systems that support extended attributes…

Got a demo ?

Tons of demo videos on my YouTube channel. Free beta available for download on my website. Links in my profile.

Re: Show HN: VectorVFS, your filesystem as a vector database

#88

Post by hadlock »

danudey wrote: post #50
3abiton wrote: post #43

Earlier quoted context omitted.

I am curious why Python, and not rust for example?

Not OP, but despite working in an all-Go shop I just wrote a utility in Python the other week and caught some flak for it. The reason I gave (which was accepted) was that the process of creating a proof of concept and iterating on it rapidly is vastly easier in Python (for me) than it is in Go. In essence, it would have taken me at least a week, possibly more, to write the program I ended up with in Golang, but it on…

Sure, but now your all-go shop now needs to support two languages, two sets of linters, ci/cd etc for a single utility. It might be faster for you but if the utility is going to be used for more than a couple of weeks now it's a real hassle to get a go developer to make sure they have the right version of the interpeter, remember all the ins and outs of python etc.

Re: Show HN: VectorVFS, your filesystem as a vector database

#89

Post by Ericson2314 »

foobiekr wrote: post #81
Ericson2314 wrote: post #61

Earlier quoted context omitted.

Yes I agree, that assertion doesn't pass muster. Mature database implementations also bypass a lot of kernel machinary to get closer to the underlying block devices. The layering of DB on top of FS is a failure.

You are confusing that databases implement their own filesystem equivalent functionality in an application-specific way with the idea that FS's can or should be databases.

I am not confusing any such thing. You need to define "database" such that "file system" doesn't include it.

Common usage does this by convention, but that's just sloppy thinking and populist extentional definitining. I posit that any rigorous, thought-out, not overfit intentional definition of a database will, as a matter of course, also include file systems.

Re: Show HN: VectorVFS, your filesystem as a vector database

#90

Post by Ericson2314 »

runlaszlorun wrote: post #30
Ericson2314 wrote: post #10

The idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.

I've heard this mentioned a couple times but what would this look like functionality wise? A single "files" table with columns? Different tables for different categories of files? FTS? Something else?

See the other comments. The point is not a specific new interface, but a separation of concerns, and leveling the playing field.

I'll try to do an example. The kernel doesn't currently know about SQL. Instead, you e.g. connect to a socket, and start talking to postgres. Imagine if FS stuff was the same thing: you connect to a socket, and then issue various command to read and write files. Ignore perf for a moment, it works right?

Now, one counter-argument might be "hold up, what is this socket you need to connect to, isn't that part of a file system? Is there now an all-userspace inner filesystem, still kernel-supported 'meta filesystem'?" Well, the answer to that is maybe the Unix idea of making communication channels like pipes and (to a lesser extent) sockets, was a bad idea. Or rather, there may be nothing wrong with saying a directory can have a child which may be such a communication channel, but there is a problem with saying that every such communication channel should live inside some directory.

Post reply on HN