Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your File System Is Already A Graph Database

rumproarious.com

Post reply on HN
Print view View original

61–70 of 85 posts

Re: Your File System Is Already A Graph Database

#61

Post by kenforthewin »

iwontberude wrote: post #58
kenforthewin wrote: post #43

I keep harping on this, but the question is not "can you use your filesystem as a graph database" - of course you can - but whether this performs better or worse than a vector database approach, especially at scale. The premise of Atomic, the knowledge base project I'm currently working on, is that there is still significant value in vectors, even in an agentic context. https://github.com/kenforthewin/atomic

Oh neat! Whenever I was working on RAG proof of concepts vector databases seemed to generate noisiest outputs that happened to include my information from my chunks but it was unable to draw reasonable contextual associations. I swap RAG out with a web search tool, all of a sudden the quality goes way up. Is RAG ever going to be easier to hold or should lay people like me just stay moving on?

I think agentic RAG still has its place. a hybrid semantic/keyword search tool in addition to other research tools outperforms the baseline in my experience.

Re: Your File System Is Already A Graph Database

#62

Post by lamasery »

zadikian wrote: post #47
lamasery wrote: post #28

Earlier quoted context omitted.

Neo4j looooooves the "if you think about it, everything is graphs!" marketing maneuver. They (their marketing department) were the very first thing I thought of when I read this headline.

"Everything is graphs, so let's use a graph DBMS for anything" is a classic blunder

I've seen it work to sell their product to managers who definitely should have gone with something else, so I get why they do it. It works.

Re: Your File System Is Already A Graph Database

#64

Post by estetlinus »

I am more curious on the note taking. How do you ingest data here? Export from slack via LLM:s? Store it in GitHub?

My “knowledge” is spread out on various SaaS (Google, slack, linear, notion, etc). I don’t see how I can centralize my “knowledge” without a lot of manual labour.

Re: Your File System Is Already A Graph Database

#65

Post by rzzzt »

volemo wrote: post #29
stingraycharles wrote: post #22

Using the same logic, a key/value database is also a graph database? Isn’t the biggest benefit of graph databases the indexing and additional query constructs they support, like shortest path finding and whatnot?

I think the confusion stems from the fact that we call a database what is really a database management system.

I'd just like to interject for a moment. What you're refering to as a database, is in fact a database management system, or as I've recently taken to calling it, database plus management system.

Re: Your File System Is Already A Graph Database

#66

Post by kesor »

So you have some folders with markdown files ... which are insanely hard to query without a tool ... impossible to traverse via their relationships ... and you call that a graph database? WHAT?!

Clicked the link expecting to see some tool or method that actually allows graph-like queries and traversals on files in a file system, all I found was some rant about someone on the internet being wrong.

Waste of time.

Re: Your File System Is Already A Graph Database

#67

Post by LocalPCGuy »

estetlinus wrote: post #64

I am more curious on the note taking. How do you ingest data here? Export from slack via LLM:s? Store it in GitHub? My “knowledge” is spread out on various SaaS (Google, slack, linear, notion, etc). I don’t see how I can centralize my “knowledge” without a lot of manual labour.

Unless you're forced into using certain tool (work, etc), start by standardizing on a single tool. That's one reason a lot of people like Obsidian, but there are plenty of similar tools, or you can just write markdown in your editor of choice. Then set of some sort of sync so you have it everywhere you are (mobile can be a bit tricky for some set-ups) and commit to using that method as much as possible for your notes.

You may want to do as described and link to Slack messages (etc), but just remember any external link should be treated as ephemeral. You may not have access to the Slack anymore, for example. That may mean you don't need that note either, or it may mean you lost access to a node on your knowledge graph, you have to determine whether that matters.

By starting now, at least everything going forward is captured in a way you can both own and utilize it. Then it may be a bit of a pain and some manual work to get existing notes into your tool of choice, but you can determine what needs to be in there from other tools as you go forward.

Re: Your File System Is Already A Graph Database

#69

Post by zadikian »

embedding-shape wrote: post #48
zadikian wrote: post #44

On the other hand, I get why cloud drive users completely disregard file structure and search everything. Two files usually don't have the same name unless you're laying it out programmatically like this. I use dir trees for code ofc, but everything else is flat in my ~/Documents. Deep inside a project dir, feels like some the ease of LLMs is just not having to cd into the correct directory, but you shouldn't need an…

> I use dir trees for code ofc, but everything else is flat in my ~/Documents. Which is great, but on all major OSes you'd eventually hit performance issues with flat directories like this. Might not be an issue in month one, or even year one, but after 10 years of note taking/journaling that approach will show the issue with large flat directories. So eventually you'd need to shard it somehow, so might as well start…

If it's just performance, cd ~/Documents && mkdir old && mv ./* old/ (or today's date instead of old). I actually have that layout on one PC.

If real organization is needed, seems like that'd be easier in hindsight than having foresight

Re: Your File System Is Already A Graph Database

#70

Post by mswphd »

embedding-shape wrote: post #40
SeanLang wrote: post #35

Earlier quoted context omitted.

Couldn't you create synthetic data based on your entries using local models? Or would that defeat the purpose of fine tuning it?

Yeah, I suppose, but how do I get sufficiently high quality synthetic data without sending the original data to OpenAI/Anthropic, or by using local models when none of them seem strong enough to be able to generate that "sufficiently high quality synthetic data" in the first place?

you could do something like rent GPU time yourself, and use it to run a higher-quality local model (e.g. one of the Chinese "close to frontier" ones). Not guaranteed to preserve privacy of course, but it at least avoids directly sending the data to OpenAI/Anthropic.
Post reply on HN