Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: VectorVFS, your filesystem as a vector database

vectorvfs.readthedocs.io

Post reply on HN
Print view View original

21–30 of 150 posts

Re: Show HN: VectorVFS, your filesystem as a vector database

#21

Post by adolph »

foobiekr wrote: post #16
Ericson2314 wrote: post #10

The idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.

Every single time this has been tried it has gone wrong, but sure. Almost all of the operations done on actual filesystems are not database like, they are close to the underlying hardware for practical reasons. If you want a database view, add one in an upper layer.

> they are close to the underlying hardware for practical reasons

Could you provide reference information to support this background assertion? I'm not totally familiar with filesystems under the hood, but at this point doesn't storage hardware maintain an electrical representation relatively independent from the logical given things like wear leveling?

Re: Show HN: VectorVFS, your filesystem as a vector database

#22

Post by esafak »

malcolmgreaves wrote: post #2

Fun idea storing embeddings in inodes! Very clever! I want to point out that this isn’t suitable for any kind of actual things you’d use a vector database for. There’s no notion of a search index. It’s always a O(N) linear search through all of your files: https://github.com/perone/vectorvfs/blob/main/vectorvfs/cli.... Still, fun idea :)

thanks for saving readers time. If so this is not a viable tool for production.

Re: Show HN: VectorVFS, your filesystem as a vector database

#23

Post by hdevalence »

qwertox wrote: post #13
Ericson2314 wrote: post #10

The idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.

I can't agree with this. I like it that I can have all these tools which work with files and are tools which are not db-oriented, and the fact that there are different filesystems for different scenarios, that I can sandwich LVM between a FS and the block device. That /proc/ can pretend to be a FS because else we'd possibly end up with something like the Windows Registry for these operations, only managed through a d…

Yes, I would

Re: Show HN: VectorVFS, your filesystem as a vector database

#26

Post by 90s_dev »

qwertox wrote: post #13
Ericson2314 wrote: post #10

The idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.

I can't agree with this. I like it that I can have all these tools which work with files and are tools which are not db-oriented, and the fact that there are different filesystems for different scenarios, that I can sandwich LVM between a FS and the block device. That /proc/ can pretend to be a FS because else we'd possibly end up with something like the Windows Registry for these operations, only managed through a d…

> Would you store all your ~/ in something like SQLite database?

Actually yeah that sounds pretty good.

For Desktop/Finder/Explorer you'd just need a nice UI.

Searching Documents/projects/etc would be the same just maybe faster?

All the arbitrary stuff like ~/.npm/**/* would stop cluttering up my ls -la in ~ and could be stored in their own tables whose names I genuinely don't care about. (This was the dream of ~/Library, no?)

[edit] Ooooh, I get it now. This doesn't solve namespacing or traversal.

Re: Show HN: VectorVFS, your filesystem as a vector database

#27

Post by tzury »

I’ve found that starting with a plain old filesystem often outperforms fancy services - just as the Unix philosophy (“everything is a file” [1]) has preached for decades [2].

When BigQuery was still in alpha I had to ingest ~15 billion HTTP requests a day (headers, bodies, and metadata). None of the official tooling was ready, so I wrote a tiny bash script that:

    1. uploaded the raw logs to Cloud Storage, and
    2. tracked state with three folders: `pending/`, `processing/`, `done/`.
A cron job cycled through those directories and quietly pushed petabytes every week without dropping a byte. Later, Google’s own pipelines—and third-party stacks like Logstash—never matched that script’s throughput or reliability.

Lesson: reach for the filesystem first; add services only once you’ve proven you actually need them.

[1] https://en.wikipedia.org/wiki/Everything_is_a_file [2] https://en.wikipedia.org/wiki/Unix_philosophy

Re: Show HN: VectorVFS, your filesystem as a vector database

#28

Post by dominicq »

tzury wrote: post #27

I’ve found that starting with a plain old filesystem often outperforms fancy services - just as the Unix philosophy (“everything is a file” [1]) has preached for decades [2]. When BigQuery was still in alpha I had to ingest ~15 billion HTTP requests a day (headers, bodies, and metadata). None of the official tooling was ready, so I wrote a tiny bash script that: 1. uploaded the raw logs to Cloud Storage, and 2. track…

Can you say more about the use case? What problem were you solving? How did it work exactly? Sounds interesting so I'd like to learn more.

Re: Show HN: VectorVFS, your filesystem as a vector database

#29

Post by bullen »

I did something similar, but I use these EXT4 requirements:

  - hard links (only tar works for backup)
  - small file size (or inodes run out before disk space)
http://root.rupy.se

It's very useful for global distributed real-time data that don't need the P in CAP for writes.

(no new data can be created if one node is offline = you can login, but not register)

Re: Show HN: VectorVFS, your filesystem as a vector database

#30

Post by runlaszlorun »

Ericson2314 wrote: post #10

The idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.

I've heard this mentioned a couple times but what would this look like functionality wise? A single "files" table with columns? Different tables for different categories of files? FTS? Something else?
Post reply on HN