Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grafeo – A fast, lean, embeddable graph database built in Rust

grafeo.dev

Post reply on HN
Print view View original

91–100 of 106 posts

Re: Grafeo – A fast, lean, embeddable graph database built in Rust

#91

Post by adsharma »

zozbot234 wrote: post #68
SkyPuncher wrote: post #64

Every time I look at graph databases, I just cannot figure out what problem they're solving. Particularly in an LLM based world. Don't get me wrong, graphs have interesting properties and there's something intriguing out these dynamic, open ended queries. But, what features/products/customer journeys are people building with a graph DB. Every time I explore, I end up back at "yea, but a standard DB will do 90% of thi…

A standard DB ala Postgres will be a perfectly functional graph database unless you're doing very specialized network analysis queries, which is not what most of these "knowledge graph" databases are being used for. It's only querying and data modeling that's a bit fiddly (expressing the "graph" structure using SQL) and that's being improved by the new Property Graph Query (PGQ) in the latest SQL standards.

This is the same topic I had an intense argument with my coworkers at the company formerly called FB a decade ago. There is a belief that most joins are 1-2 deep. And that many hop queries with reasoning are rare and non-existent.

I wonder how you reconcile the demand for LLMs with multihop reasoning with the statement above.

I think a lot what is stated here is how things work today and where established companies operate.

The contradictions in their positions are plain and simple.

Re: Grafeo – A fast, lean, embeddable graph database built in Rust

#92

Post by IshKebab »

macintux wrote: post #65
IshKebab wrote: post #54

Earlier quoted context omitted.

Because Rust is an excellent language that pushes you into the "pit of success", and consequently software written in Rust tends to be fast, robust and easy to deploy. There's no big mystery. No conspiracy or organised evangelism. Rust is just really good.

Worth noting that “robust” and “correct” are orthogonal. Graph databases (well, any database) seem like an area where correctness particularly matters, and I doubt Rust gives any meaningful advantage there.

They absolutely are not orthogonal. They are closely related. In any case, Rust improves both.

> I doubt Rust gives any meaningful advantage there.

Advantage over what? Haskell & OCaml? Maybe not. C++ or Python? Absolutely. Its type system is far stronger than those, and its APIs are much better designed and harder to misuse.

Re: Grafeo – A fast, lean, embeddable graph database built in Rust

#93

Post by zozbot234 »

adsharma wrote: post #91
zozbot234 wrote: post #68

Earlier quoted context omitted.

A standard DB ala Postgres will be a perfectly functional graph database unless you're doing very specialized network analysis queries, which is not what most of these "knowledge graph" databases are being used for. It's only querying and data modeling that's a bit fiddly (expressing the "graph" structure using SQL) and that's being improved by the new Property Graph Query (PGQ) in the latest SQL standards.

This is the same topic I had an intense argument with my coworkers at the company formerly called FB a decade ago. There is a belief that most joins are 1-2 deep. And that many hop queries with reasoning are rare and non-existent. I wonder how you reconcile the demand for LLMs with multihop reasoning with the statement above. I think a lot what is stated here is how things work today and where established companies o…

There are worst-case optimal algorithms for multi-way and multi-hop joins. This does not require giving up the relational model.

Re: Grafeo – A fast, lean, embeddable graph database built in Rust

#94

Post by adsharma »

zozbot234 wrote: post #93
adsharma wrote: post #91

Earlier quoted context omitted.

This is the same topic I had an intense argument with my coworkers at the company formerly called FB a decade ago. There is a belief that most joins are 1-2 deep. And that many hop queries with reasoning are rare and non-existent. I wonder how you reconcile the demand for LLMs with multihop reasoning with the statement above. I think a lot what is stated here is how things work today and where established companies o…

There are worst-case optimal algorithms for multi-way and multi-hop joins. This does not require giving up the relational model.

I maintain LadybugDB which implements WCOJ (inherited from the KuzuDB days). So I don't disagree with the idea. Just that it's a graph database with relational internals and some internal warts that makes it hard to compose queries. Working on fixing them.

https://github.com/LadybugDB/ladybug/discussions/204#discuss...

Re: Grafeo – A fast, lean, embeddable graph database built in Rust

#95

Post by adsharma »

zozbot234 wrote: post #93
adsharma wrote: post #91

Earlier quoted context omitted.

This is the same topic I had an intense argument with my coworkers at the company formerly called FB a decade ago. There is a belief that most joins are 1-2 deep. And that many hop queries with reasoning are rare and non-existent. I wonder how you reconcile the demand for LLMs with multihop reasoning with the statement above. I think a lot what is stated here is how things work today and where established companies o…

There are worst-case optimal algorithms for multi-way and multi-hop joins. This does not require giving up the relational model.

Also an important test is the check on whether it's WCOJ on top of relational storage or is the compressed sparse row (CSR) actually persisted to disk. The PGQ implementations don't.

There are second order optimizations that LLMs logically implement that CSR implementing DBs don't. With sufficient funding, we'll be able to pursue those as well.

Re: Grafeo – A fast, lean, embeddable graph database built in Rust

#97

Post by StevenBtw »

Aurornis wrote: post #4

Does anyone have any experience with this DB? Or context about where it came from? From the commit history it's obvious that this is an AI coded project. It was started a few months ago, 99% of commits are from 1 contributor, and that 1 contributor has some times committed 100,000 lines of code per week. (EDIT: 200,000 lines of code in the first week) I'm not anti-LLM, but I've done enough AI coding to know that one…

Hi, I'm the one building grafeo, I have no idea why it is being posted everywhere. But I can probably answer your questions.

The first version was largely a (slightly rearchitected) port of a local graph database I had been building called graphos. Most of the engine and core are handwritten, so are the python bindings and conformance tests. The rest is indeed largely AI generated, so is the documentation (Mkdocs). The AI generated parts are curated and validated, although it's not up to par for a production release yet.

This is not a resume portfolio project and in no way related to my day job. I started writing grafeo(then graphos) out of frustration with Neo4j and being inspired by some discussions about database internals with Hännes from duckdb at a conference. I tried ladybug, but found memory usage insanely high and was sure I could do better. Anyone looking for an embedded battle tested graph database should probably still look at ladybug though. Grafeo is not that mature yet.

And to be honest I also have no real plans with grafeo, I am using it myself for now and am very happy with it, but that's n=1. It's fully free and open source and contributors are very welcome, but its also not yet fully where I would want it to be, hence the beta status. I have no commercial interest, but had a lot of fun pouring multiple hundreds of hours in and creating something that I enjoy using myself.

Hope that clarifies some things!

Re: Grafeo – A fast, lean, embeddable graph database built in Rust

#98

Post by adsharma »

Sytten wrote: post #80
adsharma wrote: post #3

There are 25 graph databases all going me too in the AI/LLM driven cycle. Writing it in Rust gets visibility because of the popularity of the language on HN. Here's why we are not doing it for LadybugDB. Would love to explore a more gradual/incremental path. Also focusing on just one query language: strongly typed cypher. https://github.com/LadybugDB/ladybug/discussions/141

I really wish people would stop using the language as an argument and that commenter would also move on to a more interesting debate. In your discussion the first comment from an ex kuzu dev made an excellent point that rust for databases in an excellent language to ship faster with confidence while reducing real problems of concurrency and corruption. At some point it becomes intellectual dishonesty to dismiss a lan…

I didn't dismiss the language. I called it a north star. Rust is still the best option if you desire memory safety.

But rewriting a complex working piece of software in Rust is not trivial. Having an incremental path (where only parts are rewritten in Rust and compatible with C++ code) would be a good path to get there.

Also open to new code and extensions getting written in Rust.

Re: Grafeo – A fast, lean, embeddable graph database built in Rust

#99

Post by zozbot234 »

adsharma wrote: post #95
zozbot234 wrote: post #93

Earlier quoted context omitted.

There are worst-case optimal algorithms for multi-way and multi-hop joins. This does not require giving up the relational model.

Also an important test is the check on whether it's WCOJ on top of relational storage or is the compressed sparse row (CSR) actually persisted to disk. The PGQ implementations don't. There are second order optimizations that LLMs logically implement that CSR implementing DBs don't. With sufficient funding, we'll be able to pursue those as well.

CSR is an array-based trie hence very costly to update. It can serve as an index for parts of the graph that basically will almost never change, but not otherwise.

Re: Grafeo – A fast, lean, embeddable graph database built in Rust

#100

Post by adsharma »

zozbot234 wrote: post #99
adsharma wrote: post #95

Earlier quoted context omitted.

Also an important test is the check on whether it's WCOJ on top of relational storage or is the compressed sparse row (CSR) actually persisted to disk. The PGQ implementations don't. There are second order optimizations that LLMs logically implement that CSR implementing DBs don't. With sufficient funding, we'll be able to pursue those as well.

CSR is an array-based trie hence very costly to update. It can serve as an index for parts of the graph that basically will almost never change, but not otherwise.

Makes it a good match for columnar databases which already operate on the read-only, read-mostly part of the spectrum.

Perhaps people can invent LSM like structures on top of them.

But at least establish that CSR on disk is a basic requirement before you claim that you're a legit graph database.

Post reply on HN