Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: VectorVFS, your filesystem as a vector database

vectorvfs.readthedocs.io

Post reply on HN
Print view View original

101–110 of 150 posts

Re: Show HN: VectorVFS, your filesystem as a vector database

#101

Post by WalterGR »

p_ing wrote: post #80
didgetmaster wrote: post #71

Earlier quoted context omitted.

I think that is one of the main reason it failed to launch. It was just too easy for the metadata stored in the separate database to become out of sync with the actual file data. Microsoft saw the tech support nightmare this could generate, and abandoned the project.

It was abandoned due to The Cloud. There was no need for WinFS as a tech when you could store everything in The Cloud. It was also complex, ran poorly, and would have required developers to integrate their applications. Microsoft had long solved the problem of blobs and metadata in ESE and SharePoint's use of MS SQL for binary + metadata storage.

> it was just a SQL database that stored arbitrary data.

I mean, for some definitions of “just”, “SQL database”, and “arbitrary data.” :) It was a schematised graph database implemented on top of a slimmed-down version of SQL Server. The query language was not SQL-based.

> It was abandoned due to The Cloud.

It was discontinued circa 2007. The cloud was much less of a Thing back then. I don’t recall that factoring at all into the decision to cancel the project, though it would have been prescient.

(Disclaimer: I was on the WinFS team at Microsoft.)

Re: Show HN: VectorVFS, your filesystem as a vector database

#102

Post by foobiekr »

Ericson2314 wrote: post #92
foobiekr wrote: post #79

Earlier quoted context omitted.

BeFS wasn't a database. It had indexed queries on EAs and they had the habit of asking application files to add their indexable content to the EAs. Internally it was just a mostly-not-transactional collection of btrees. There was no query language for updating files, or even inspecting anything about a file that was not published in the EAs (or implicitly do as with adapters), there were no multi-file transactions, n…

Yeah I am talking more deep architecture, and BeOS is more notable here mostly on just the user-interface level. However, I think it is reasonable to think that with way more time and money, these things would meet up. Think about it as digging a tunnel from both sides of the mountain.

Microsoft poured at least $100M into this hole with nothing to show for it.

Re: Show HN: VectorVFS, your filesystem as a vector database

#103

Post by foobiekr »

Ericson2314 wrote: post #89
foobiekr wrote: post #81

Earlier quoted context omitted.

You are confusing that databases implement their own filesystem equivalent functionality in an application-specific way with the idea that FS's can or should be databases.

I am not confusing any such thing. You need to define "database" such that "file system" doesn't include it. Common usage does this by convention, but that's just sloppy thinking and populist extentional definitining. I posit that any rigorous, thought-out, not overfit intentional definition of a database will, as a matter of course, also include file systems.

Perhaps so, but in the expansive definition you're using, even an in-memory binary tree qualifies as a database, which makes your point meaningless.

Re: Show HN: VectorVFS, your filesystem as a vector database

#104

Post by p_ing »

WalterGR wrote: post #101
p_ing wrote: post #80

Earlier quoted context omitted.

It was abandoned due to The Cloud. There was no need for WinFS as a tech when you could store everything in The Cloud. It was also complex, ran poorly, and would have required developers to integrate their applications. Microsoft had long solved the problem of blobs and metadata in ESE and SharePoint's use of MS SQL for binary + metadata storage.

> it was just a SQL database that stored arbitrary data. I mean, for some definitions of “just”, “SQL database”, and “arbitrary data.” :) It was a schematised graph database implemented on top of a slimmed-down version of SQL Server. The query language was not SQL-based. > It was abandoned due to The Cloud. It was discontinued circa 2007. The cloud was much less of a Thing back then. I don’t recall that factoring at…

Skydrive released in 2007.

But fair enough, I grabbed my Beta 1 copy from \\products; it was fun to play with. I wish they’d seen it through. Microsoft had plenty of 'slimmed down' versions of SQL Server, i.e. the CRM addin for Outlook, so that isn't quite a unique feature of WinFS.

Re: Show HN: VectorVFS, your filesystem as a vector database

#105

Post by lysp »

binarymax wrote: post #96
thirdtrigger wrote: post #93

Might be interesting to add an optional embedded Weaviate [1] with a flat-index [2] to the project. It wouldn't use external services and is fully disk-based. Would allow you to search the whole filesystem (about 1.5kb per file (384 dimensions) which would be added to the metadata as well). 1. https://weaviate.io/developers/weaviate/installation/embedde... 2. https://weaviate.io/developers/academy/py/vector_index/fla…

Why weaviate and not FAISS? The latter is faster and lighter.

I think they are associated with the project

Re: Show HN: VectorVFS, your filesystem as a vector database

#106

Post by PeterZaitsev »

I think comparing it to Vector Database is confusing as database would typically mean indexes and some sort of query support.

Storing Embeddings with File is interesting concept... we already do it for some file formats (ie EXIF), where this one is generalized... yet you would need to have some actual database to load this data into to process at scale.

Another issue I see is support for different models and embedding formats to make this data really portable - like I can take my file drop it into any system and its embedding "seamlessly" integrates

Re: Show HN: VectorVFS, your filesystem as a vector database

#107

Post by aforwardslash »

p_ing wrote: post #69
pseudosavant wrote: post #52

This immediately made me nostalgic for BeOS's BeFS or Windows Longhorn's WinFS database filesystems, and how this kind of thing would have fit them perfect. So much cool stuff you could do with vectors for everything. Smart folders that include files for a project based on a description of the project. Show me all of my config files for appXYZ. Images of a black dog at the beach. At the OS-level for any other app to…

WinFS wasn't a file system laid down on hardware, it was just a SQL database that stored arbitrary data.

Most modern filesystems aren't really "laid down on hardware", but on top of a volume manager; thats how both attached disks (think SAN or EBS volumes) and RAID works. Most filesystems behave exactly like an optimized database index, but lacking the extra fields; also, some databases support raw devices directly(using mapped volumes), no need for filesystems. The line between both is mostly a blur.

Re: Show HN: VectorVFS, your filesystem as a vector database

#108

Post by Ericson2314 »

foobiekr wrote: post #102
Ericson2314 wrote: post #92

Earlier quoted context omitted.

Yeah I am talking more deep architecture, and BeOS is more notable here mostly on just the user-interface level. However, I think it is reasonable to think that with way more time and money, these things would meet up. Think about it as digging a tunnel from both sides of the mountain.

Microsoft poured at least $100M into this hole with nothing to show for it.

That doesn't disprove anything for me. It just says POSIX DOS lowest common denominator network effects are a hell of a drug.

Whenever we're talking about interfaces, coordination success or failure is the name of the game.

Re: Show HN: VectorVFS, your filesystem as a vector database

#109

Post by aforwardslash »

didgetmaster wrote: post #71
p_ing wrote: post #69

Earlier quoted context omitted.

WinFS wasn't a file system laid down on hardware, it was just a SQL database that stored arbitrary data.

I think that is one of the main reason it failed to launch. It was just too easy for the metadata stored in the separate database to become out of sync with the actual file data. Microsoft saw the tech support nightmare this could generate, and abandoned the project.

It failed because it was slow and provided no obvious benefit to the average user, other than wasting disk space. There is a fair amount of tricks and optimizations in NTFS to balance speed of access and shadow copy mechanisms, in a pre-SSD era, where the average random throughput of a consumer-grade SATA disk was single digit MB/s

Re: Show HN: VectorVFS, your filesystem as a vector database

#110

Post by Ericson2314 »

mrlongroots wrote: post #68
adolph wrote: post #21

Earlier quoted context omitted.

> they are close to the underlying hardware for practical reasons Could you provide reference information to support this background assertion? I'm not totally familiar with filesystems under the hood, but at this point doesn't storage hardware maintain an electrical representation relatively independent from the logical given things like wear leveling?

Some examples off the top of my head: - You can reason about block offsets. If your writes are 512B-aligned, you can be ensured minimal write amplification. - If your writes are append-only, log-structured, that makes SSD compaction a lot more straightforward - No caching guarantees by default. Again, even SSDs cache writes. Block writes are not atomic even with SSDs. The only way to guarantee atomicity is via write-…

Is file alignment on disk guaranteed, or does it depend on the file system?

The NVMe layer is not the same as the POSIX filesystem, there is no reason we need to throw that as part of knocking the POSIX filesystem off it's privileged position.

Overall you are talking about individual files, but remember what really distinguishes the filesystem is directories. Other database, even relational ones, can have binary blob "leaf data" with the properties you speak about.

Post reply on HN