Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: VectorVFS, your filesystem as a vector database

vectorvfs.readthedocs.io

Post reply on HN
Print view View original

111–120 of 150 posts

Re: Show HN: VectorVFS, your filesystem as a vector database

#111

Post by Ericson2314 »

foobiekr wrote: post #103
Ericson2314 wrote: post #89

Earlier quoted context omitted.

I am not confusing any such thing. You need to define "database" such that "file system" doesn't include it. Common usage does this by convention, but that's just sloppy thinking and populist extentional definitining. I posit that any rigorous, thought-out, not overfit intentional definition of a database will, as a matter of course, also include file systems.

Perhaps so, but in the expansive definition you're using, even an in-memory binary tree qualifies as a database, which makes your point meaningless.

I'm OK including that — tmpfs is similar, but we can easily exclude that by requiring persistence. The intentional definition doesn't need to be expansive the point of being useless!

Re: Show HN: VectorVFS, your filesystem as a vector database

#112

Post by aforwardslash »

pseudosavant wrote: post #75
didgetmaster wrote: post #71

Earlier quoted context omitted.

I think that is one of the main reason it failed to launch. It was just too easy for the metadata stored in the separate database to become out of sync with the actual file data. Microsoft saw the tech support nightmare this could generate, and abandoned the project.

They just weren't able to pull it off for whatever reason. I actually ran BeOS as my daily driver for quite a while (way) back in the day. BeFS was genuinely amazing, and not something I've seen replicated elsewhere yet. There hasn't really been anything interesting done in filesystems used by users on devices in a really long time.

BeOS was mostly a single-user system, and filesystem was optimized for that use case. Having to consider that an opened file for reading may be modified by a remote system running with a different user complicates things, so while I do agree that BeFS was quite ahead of his time, it focused on a somewhat specific use case; At the time, you wouldn't expect that you could eg. correctly backup your email client data without shutting down the application.

Re: Show HN: VectorVFS, your filesystem as a vector database

#113

Post by b0a04gl »

perone wrote: post #56
b0a04gl wrote: post #14

If VectorVFS obscures retrieval logic behind opaque embeddings, how do users debug why a file surfaced—or worse, why one didn’t?

Hi, not sure if I understood what you meant by opaque embeddings as well, but the reason why files surface or not is due to the similarity score (which is basically the dot product of embeddings).

[dead]

Re: Show HN: VectorVFS, your filesystem as a vector database

#114

Post by aforwardslash »

mrlongroots wrote: post #68
adolph wrote: post #21

Earlier quoted context omitted.

> they are close to the underlying hardware for practical reasons Could you provide reference information to support this background assertion? I'm not totally familiar with filesystems under the hood, but at this point doesn't storage hardware maintain an electrical representation relatively independent from the logical given things like wear leveling?

Some examples off the top of my head: - You can reason about block offsets. If your writes are 512B-aligned, you can be ensured minimal write amplification. - If your writes are append-only, log-structured, that makes SSD compaction a lot more straightforward - No caching guarantees by default. Again, even SSDs cache writes. Block writes are not atomic even with SSDs. The only way to guarantee atomicity is via write-…

> You can reason about block offsets. If your writes are 512B-aligned, you can be ensured minimal write amplification.

Not all devices use 512 byte sectors, an that is mostly a relic from low-density spinning rust;

> If your writes are append-only, log-structured, that makes SSD compaction a lot more straightforward

Hum, no. Your volume may be a sparse file on SAN system; in fact that is often the case in cloud environments; also, most cached RAID controllers may have different behaviours on this - unless you know exactly what your targeting, you're shooting blind.

> No caching guarantees by default. Again, even SSDs cache writes. Block writes are not atomic even with SSDs. The only way to guarantee atomicity is via write-ahead logs.

Not even that way. Most server-grade controllers (with battery) will ack an fsync immediately, even if the data is not on disk yet.

> The NVMe layer exposes async submission/completion queues, to control the io_depth the device is subjected to, which is essential to get max perf from modern NVMe SSDs.

Thats storage domain, not application domain. In most cloud systems, you have the choice of using direct attached storage (usually with a proper controller, so what is exposed is actually the controller features, not the individual nvme queue), or SAN storage - a sparse file on a filesystem on a system that is at the end of a tcp endpoint. One of those provides easy backups, redundancy, high availability and snapshots, and the other one you roll your own.

Re: Show HN: VectorVFS, your filesystem as a vector database

#115

Post by aforwardslash »

foobiekr wrote: post #103
Ericson2314 wrote: post #89

Earlier quoted context omitted.

I am not confusing any such thing. You need to define "database" such that "file system" doesn't include it. Common usage does this by convention, but that's just sloppy thinking and populist extentional definitining. I posit that any rigorous, thought-out, not overfit intentional definition of a database will, as a matter of course, also include file systems.

Perhaps so, but in the expansive definition you're using, even an in-memory binary tree qualifies as a database, which makes your point meaningless.

Nitpick: An in-memory hash table can be a filesystem :)

Re: Show HN: VectorVFS, your filesystem as a vector database

#116

Post by foobiekr »

Ericson2314 wrote: post #108
foobiekr wrote: post #102

Earlier quoted context omitted.

Microsoft poured at least $100M into this hole with nothing to show for it.

That doesn't disprove anything for me. It just says POSIX DOS lowest common denominator network effects are a hell of a drug. Whenever we're talking about interfaces, coordination success or failure is the name of the game.

What problem do you think a DB-as-filesystem solves? The only obvious one that makes any sense at all is cross-file transactions.

Re: Show HN: VectorVFS, your filesystem as a vector database

#117

Post by user3939382 »

cryptonector wrote: post #84
sunshine-o wrote: post #62

Earlier quoted context omitted.

Absolutely. I would add that filesystems are superior to data formats (XML, JSON, YAML, TOML) for many use cases such as configuration or just storing data. - Hierarchy are dirs, - Keys are file names, - Value is the content of the file. - Other metadata are in hidden files It will work forever, you can leverage ZFS, Git, rsync, syncthing much better. If you want, a fancy shells like Nushell will bring the experience…

Except that now when you do need a tool like XSLT/XPath, jq, or yq, now you need bash. I use bash lots, but still I'd rather use a better language, like the ones you listed. I'm being slightly hypocritical because I've made plenty of use of the filesystem as a configuration store. In code it's quite easy to stat one path relative to a directory, or open it and read it, so it's very tempting.

You don’t need bash to traverse a file system, are you saying something else?

Re: Show HN: VectorVFS, your filesystem as a vector database

#118

Post by mrlongroots »

Ericson2314 wrote: post #110
mrlongroots wrote: post #68

Earlier quoted context omitted.

Some examples off the top of my head: - You can reason about block offsets. If your writes are 512B-aligned, you can be ensured minimal write amplification. - If your writes are append-only, log-structured, that makes SSD compaction a lot more straightforward - No caching guarantees by default. Again, even SSDs cache writes. Block writes are not atomic even with SSDs. The only way to guarantee atomicity is via write-…

Is file alignment on disk guaranteed, or does it depend on the file system? The NVMe layer is not the same as the POSIX filesystem, there is no reason we need to throw that as part of knocking the POSIX filesystem off it's privileged position. Overall you are talking about individual files, but remember what really distinguishes the filesystem is directories. Other database, even relational ones, can have binary blob…

> Is file alignment on disk guaranteed, or does it depend on the file system?

I think "guaranteed" is too strong a word given the number of filesystems and flags out there, but "largely" you get aligned I/O.

> The NVMe layer is not the same as the POSIX filesystem, there is no reason we need to throw that as part of knocking the POSIX filesystem off it's privileged position.

I'd say that the POSIX filesystem lives in an ecosystem that makes leveraging NVMe layer characteristics a viable option. More along with the next point.

> Overall you are talking about individual files, but remember what really distinguishes the filesystem is directories. Other database, even relational ones, can have binary blob "leaf data" with the properties you speak about.

I think regardless of how you use a database, your interface is declarative. You always say "update this row" vs "fseek to offset 1048496 and fwrite 128 bytes and then fsync the page cache". Something needs to do the translation from "update this row" to the latter, and that layer will always be closer to hardware.

Re: Show HN: VectorVFS, your filesystem as a vector database

#120

Post by mrlongroots »

aforwardslash wrote: post #114
mrlongroots wrote: post #68

Earlier quoted context omitted.

Some examples off the top of my head: - You can reason about block offsets. If your writes are 512B-aligned, you can be ensured minimal write amplification. - If your writes are append-only, log-structured, that makes SSD compaction a lot more straightforward - No caching guarantees by default. Again, even SSDs cache writes. Block writes are not atomic even with SSDs. The only way to guarantee atomicity is via write-…

> You can reason about block offsets. If your writes are 512B-aligned, you can be ensured minimal write amplification. Not all devices use 512 byte sectors, an that is mostly a relic from low-density spinning rust; > If your writes are append-only, log-structured, that makes SSD compaction a lot more straightforward Hum, no. Your volume may be a sparse file on SAN system; in fact that is often the case in cloud envir…

I'm not sure how any of these negate the broader point: filesystems provide a lower-level interface to the underlying block device than a database.

To say that that's not true would require more than cherry-picking examples of where some fileystem assumption may be tenuous, it would require demonstrating how a DBMS can do better.

> Not all devices use 512 byte sectors

4K then :). Files are block-aligned, and the predominant block size changed once in 40 years from 512B to 4K.

> Hum, no. Your volume may be a sparse file on SAN system

Regardless, sequential writes will always provide better write performance than random writes. With POSIX, you control this behavior directly. With a DBMS, you control it by swapping out InnoDB for RocksDB or something.

> Thats storage domain, not application domain

It is a storage domain feature accessible to an IOPS-hungry application via a modern Linux interface like io_uring. NVMe-oF would be the networked storage interface that enables that. But this is for when you want to outperform a DBMS by 200X, aligned I/O is sufficient for 10-50X. :)

Post reply on HN