Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your File System Is Already A Graph Database

rumproarious.com

Post reply on HN
Print view View original

11–20 of 85 posts

Re: Your File System Is Already A Graph Database

#11

Post by laurowyn »

itake wrote: post #10

I'm wonder though: 1. Why does AI need that folder structure? Why not a flat list of files and let the AI agent explore with BM25 / grep, etc. 2. pre-compute compression vs compute at query time. Kaparthy (and you) are recommending pre-compressing and sorting based on hard coded human abstraction opinions that may match how the data might be queried into human-friendly buckets and language. Why not just let the AI ca…

> Why does AI need that folder structure? Why not a flat list of files and let the AI agent explore with BM25 / grep, etc.

It doesn't. The human creating the files needs it, to make it easier to traverse in future as the file count grows. At 52k files, that's a horrendous list to scroll through to find the thing you're looking for. Meanwhile, an AI can just `find . -type f -exec whatever {} \;` and be able to process it however it needs. Human doesn't need to change the way they work to appease the magic rock in the box under the desk.

Re: Your File System Is Already A Graph Database

#12

Post by itake »

laurowyn wrote: post #11
itake wrote: post #10

I'm wonder though: 1. Why does AI need that folder structure? Why not a flat list of files and let the AI agent explore with BM25 / grep, etc. 2. pre-compute compression vs compute at query time. Kaparthy (and you) are recommending pre-compressing and sorting based on hard coded human abstraction opinions that may match how the data might be queried into human-friendly buckets and language. Why not just let the AI ca…

> Why does AI need that folder structure? Why not a flat list of files and let the AI agent explore with BM25 / grep, etc. It doesn't. The human creating the files needs it, to make it easier to traverse in future as the file count grows. At 52k files, that's a horrendous list to scroll through to find the thing you're looking for. Meanwhile, an AI can just `find . -type f -exec whatever {} \;` and be able to process…

> The human creating the files needs it

why? The human would just talk to the AI agent. Why would they need to scroll through that many files?

I made a similar system with 232k files (1 file might be a slack message, gitlab comment, etc). it does a decent job at answering questions with only keyword search, but I think i can have better results with RAG+BM25.

Re: Your File System Is Already A Graph Database

#13

Post by stared »

Filesystem is a tree - a particular, constrained graph. Advanced topics usually require a lot of interconnections.

Maybe it is why mind maps never spoke to me. I felt that a tree structure (or even - planar graphs) were not enough to cover any sufficiently complex topic.

Re: Your File System Is Already A Graph Database

#14

Post by laurowyn »

itake wrote: post #12
laurowyn wrote: post #11

Earlier quoted context omitted.

> Why does AI need that folder structure? Why not a flat list of files and let the AI agent explore with BM25 / grep, etc. It doesn't. The human creating the files needs it, to make it easier to traverse in future as the file count grows. At 52k files, that's a horrendous list to scroll through to find the thing you're looking for. Meanwhile, an AI can just `find . -type f -exec whatever {} \;` and be able to process…

> The human creating the files needs it why? The human would just talk to the AI agent. Why would they need to scroll through that many files? I made a similar system with 232k files (1 file might be a slack message, gitlab comment, etc). it does a decent job at answering questions with only keyword search, but I think i can have better results with RAG+BM25.

And when the system fails for whatever reason?

Just because AI exists doesn't mean we can neglect basic design principles.

If we throw everything out the window, why don't we just name every file as a hash of its content? Why bother with ASCII names at all?

Fundamentally, it's the human that needs to maintain the system and fix it when it breaks, and that becomes significantly easier if it's designed in a way a human would interact with it. Take the AI away, and you still have a perfectly reasonable data store that a human can continue using.

Re: Your File System Is Already A Graph Database

#15

Post by nutjob2 »

stared wrote: post #13

Filesystem is a tree - a particular, constrained graph. Advanced topics usually require a lot of interconnections. Maybe it is why mind maps never spoke to me. I felt that a tree structure (or even - planar graphs) were not enough to cover any sufficiently complex topic.

If it has hard or soft links, its a proper graph.

Re: Your File System Is Already A Graph Database

#16

Post by [deleted] »

itake wrote: post #10

I'm wonder though: 1. Why does AI need that folder structure? Why not a flat list of files and let the AI agent explore with BM25 / grep, etc. 2. pre-compute compression vs compute at query time. Kaparthy (and you) are recommending pre-compressing and sorting based on hard coded human abstraction opinions that may match how the data might be queried into human-friendly buckets and language. Why not just let the AI ca…

[deleted]

Re: Your File System Is Already A Graph Database

#17

Post by bullen »

Yep, my distributed JSON over HTTP database uses the ext4 binary tree for indexing: http://root.rupy.se

It can only handle 3 way multiple cross references by using 2 folders and a file now (meta) and it's very verbose on the disk (needs type=small otherwise inodes run out before disk space)... but it's incredibly fast and practially unstoppable in read uptime!

Also the simplicity in using text and the file system sort of guarantees longevity and stability even if most people like the monolithic garbled mess that is relational databases binary table formats...

Re: Your File System Is Already A Graph Database

#18

Post by appsoftware »

I created AS Notes (https://www.asnotes.io) (an extension for VS Code, Antigravity etc) partly because of this use case. It works like Obsidian, being markdown based, with wikilinks, mermaid rendering and task management. In VS Code, we have access to really good Agent harnesses and can navigate our notes and documents in a file system like manner. Further, using AGENTS.md, idea files etc we can instruct the agent how to interact, add to our notes etc. I've found working with my notes like this really useful, and provided I trim anything generated by an AI that's not going to be useful, provides an investment in the information I've gathered as the information is retained in markdown rather than getting lost in multiple chatbot UI s.

Re: Your File System Is Already A Graph Database

#19

Post by calgoo »

nutjob2 wrote: post #15
stared wrote: post #13

Filesystem is a tree - a particular, constrained graph. Advanced topics usually require a lot of interconnections. Maybe it is why mind maps never spoke to me. I felt that a tree structure (or even - planar graphs) were not enough to cover any sufficiently complex topic.

If it has hard or soft links, its a proper graph.

That what i was thinking! Instead of Wiki links, use Symlinks (i guess windows would not like it?)

Re: Your File System Is Already A Graph Database

#20

Post by zahlman »

nutjob2 wrote: post #15
stared wrote: post #13

Filesystem is a tree - a particular, constrained graph. Advanced topics usually require a lot of interconnections. Maybe it is why mind maps never spoke to me. I felt that a tree structure (or even - planar graphs) were not enough to cover any sufficiently complex topic.

If it has hard or soft links, its a proper graph.

On Linux at least, hard links can't be made to directories, except for the magic . and .. links. So this only allows for a DAG.

Symbolic links can form a graph, and you can process them as needed using readlink etc. to traverse the graph, but they'll still be considered broken if they form a cycle.

Post reply on HN