Rt
Show HN: VectorVFS, your filesystem as a vector database
51–60 of 150 posts
Re: Show HN: VectorVFS, your filesystem as a vector database
#52This immediately made me nostalgic for BeOS's BeFS or Windows Longhorn's WinFS database filesystems, and how this kind of thing would have fit them perfect. So much cool stuff you could do with vectors for everything. Smart folders that include files for a project based on a description of the project. Show me all of my config files for appXYZ. Images of a black dog at the beach. At the OS-level for any other app to easily tap into.
I'd be surprised if cloud storage services like OneDrive don't already do some kind of vector for every file you store. But an online web service isn't the same as being built into the core of the OS.
Re: Show HN: VectorVFS, your filesystem as a vector database
#53I’ve found that starting with a plain old filesystem often outperforms fancy services - just as the Unix philosophy (“everything is a file” [1]) has preached for decades [2]. When BigQuery was still in alpha I had to ingest ~15 billion HTTP requests a day (headers, bodies, and metadata). None of the official tooling was ready, so I wrote a tiny bash script that: 1. uploaded the raw logs to Cloud Storage, and 2. track…
Can you say more about the use case? What problem were you solving? How did it work exactly? Sounds interesting so I'd like to learn more.
Sure.
We were building Reblaze (started 2011), a cloud WAF / DDoS-mitigation platform. Every HTTP request—good, bad, or ugly—had to be stored for offline anomaly-detection and clustering.
Traffic profile
- Baseline: ≈ 15 B requests/day
- Under attack: the same 15 B can arrive in 2-3 hours
It was the only thing that could swallow that firehose and stay query-able minutes later — crucial when you’re under attack and your data source must not melt down.
Pipeline (all shell + cron)
Edge nodes → write JSON logs locally and a local cron push to Cloud Storage
Tiny VM with a cron loop
- Scans `pending/`, composes many small blobs into one “max-size” blob in `processing/`.
- Executes `bq load …` into the customer’s isolated dataset.
- On success, moves the blob to `done/`; on failure, drops it back to `pending/`.
That handful of `gsutil`, `bq`, and `mv` commands moved multiple petabytes a week without losing a byte. Later pipelines—Dataflow, Logstash, etc.—never matched its throughput or reliability.
Re: Show HN: VectorVFS, your filesystem as a vector database
#54Earlier quoted context omitted.
so, like magic(5)?
What is magic(5) and how is it similar to what was described?
magic(5) means `man 5 magic`: https://linux.die.net/man/5/magic
It's just a tool that can read "magic bytes" to figure out what files contains. Very different from what VectorVFS is.
Re: Show HN: VectorVFS, your filesystem as a vector database
#55The idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.
You could do a loopback network filesystem and make any user-space FS you want. That's what WSL does, and there's a Rust crate for it. Can't recall the name at all.
There is NFS and FUSE so you can write your own implementation, but you are still stuck with the interface that the kernel understands.
Re: Show HN: VectorVFS, your filesystem as a vector database
#56If VectorVFS obscures retrieval logic behind opaque embeddings, how do users debug why a file surfaced—or worse, why one didn’t?
Hi, not sure if I understood what you meant by opaque embeddings as well, but the reason why files surface or not is due to the similarity score (which is basically the dot product of embeddings).
Re: Show HN: VectorVFS, your filesystem as a vector database
#57The idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.
I’m a database guy, not an OS guy, so I agree, obviously… But what is the micro-kernel angle?
The filesystem interface is only privilaged interface because it is the kernel knows about. E.g. you can already use FUSE and NFS to roll your own FS implementations, but those do not a microkernel make, because the OS is still in the way dictating the implementation.
The safest way to put the FS on a level-playing field with other interfaces is to make the kernel not know about, just as it doesn't know about, say, SQL.
Re: Show HN: VectorVFS, your filesystem as a vector database
#58Earlier quoted context omitted.
Thanks, I'm working on implementing the commands to clean the embeddings (you can now do that with Linux xattr command-line tool). I'm supporting CPU or GPU (NVIDIA) for the encoders and it only supports Linux at the moment.
I am curious why Python, and not rust for example?
Hi, I think Rust won't bring much benefit here to be honest, the bottleneck is mainly the model and model loading. It would probably be a nightmare to load these models from Rust, I would have to use torch bindings and then convert everything from the preprocessing already in Python to Rust.
Re: Show HN: VectorVFS, your filesystem as a vector database
#59The idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.
Every single time this has been tried it has gone wrong, but sure. Almost all of the operations done on actual filesystems are not database like, they are close to the underlying hardware for practical reasons. If you want a database view, add one in an upper layer.
[deleted]
Re: Show HN: VectorVFS, your filesystem as a vector database
#60Files-as-vector stores is LanceDB's value proposition. How do you compare in performance, etc.?
This is quite different than LanceDB. In VectorVFS I'm using the inodes directly to store the embeddings, there is no external file with metadata and db, the db is your filesystem itself, that's the key difference.