It's not clear that graph-bench in "Tested with the LDBC Social Network Benchmark via graph-bench" is a benchmark that you made. It seems more robust and reliable than "we built a db and a benchmark tool, and our benchmark tool says we're the best". Just a thing to be careful about. You should just state that it's your tool and you welcome feedback to help make it so that other projects being compared are compared in their best light. Something like that might help, I don't know though it's a hard problem.
Grafeo – A fast, lean, embeddable graph database built in Rust
61–70 of 106 posts
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#62Earlier quoted context omitted.
Are you talking about Andy Pavlo bet here? https://news.ycombinator.com/item?id=29737326 Kuzu folks took some of these discussions and implemented them. SIP, ASP joins, factorized joins and WCOJ. Internally it's structured very similar to DuckDB, except for the differences noted above. DuckDB 1.5 implemented sideways information passing (SIP). And LadybugDB is bringing in support for DuckDB node tables. So the idea t…
I wasn't referring to the Pavlo bet but I would make the same one! Poor algorithm and architecture scalability is a serious bottleneck. I was part of a research program working on the fundamental computer science of high-scale graph databases ~15 years ago. Even back then we could show that the architectures you mention couldn't scale even in theory. Just about everyone has been re-hashing the same basic design for d…
What do you need one trillion edges for? Wikidata is a huge, general purpose knowledge graph and it gets away with ~1B triples, give or take.
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#63I was ready to learn more about this but I saw "written in Rust" and I literally rolled my eyes and said never mind.
Because it was explicitly advertising Rust and you can't stand the zealotry or because you hate Rust?
Because the latter is really dumb. I don't mind a software written in C, although I personally wouldn't want to write it anymore.
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#64Every time I look at graph databases, I just cannot figure out what problem they're solving. Particularly in an LLM based world.
Don't get me wrong, graphs have interesting properties and there's something intriguing out these dynamic, open ended queries. But, what features/products/customer journeys are people building with a graph DB.
Every time I explore, I end up back at "yea, but a standard DB will do 90% of this as a 10% of the effort".
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#65Why is everything "... built in Rust" trending so easily on HN?
Because Rust is an excellent language that pushes you into the "pit of success", and consequently software written in Rust tends to be fast, robust and easy to deploy. There's no big mystery. No conspiracy or organised evangelism. Rust is just really good.
Worth noting that “robust” and “correct” are orthogonal. Graph databases (well, any database) seem like an area where correctness particularly matters, and I doubt Rust gives any meaningful advantage there.
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#66Every time I look at graph databases, I just cannot figure out what problem they're solving. Particularly in an LLM based world. Don't get me wrong, graphs have interesting properties and there's something intriguing out these dynamic, open ended queries. But, what features/products/customer journeys are people building with a graph DB. Every time I explore, I end up back at "yea, but a standard DB will do 90% of thi…
For starters, LLMs themselves are a graph database with probabilistic edge traversal.
Some apps want it to be deterministic.
I'm surprised this question comes up so often.
It's mainly from the vector embedding camp, who rightfully observe that vector + keyword search gets you to 70-80% on evals. What is all this hype about graphs for the last 20-30%?
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#67[flagged]
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#68Every time I look at graph databases, I just cannot figure out what problem they're solving. Particularly in an LLM based world. Don't get me wrong, graphs have interesting properties and there's something intriguing out these dynamic, open ended queries. But, what features/products/customer journeys are people building with a graph DB. Every time I explore, I end up back at "yea, but a standard DB will do 90% of thi…
A standard DB ala Postgres will be a perfectly functional graph database unless you're doing very specialized network analysis queries, which is not what most of these "knowledge graph" databases are being used for. It's only querying and data modeling that's a bit fiddly (expressing the "graph" structure using SQL) and that's being improved by the new Property Graph Query (PGQ) in the latest SQL standards.
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#69Earlier quoted context omitted.
I wasn't referring to the Pavlo bet but I would make the same one! Poor algorithm and architecture scalability is a serious bottleneck. I was part of a research program working on the fundamental computer science of high-scale graph databases ~15 years ago. Even back then we could show that the architectures you mention couldn't scale even in theory. Just about everyone has been re-hashing the same basic design for d…
What do you need one trillion edges for? Wikidata is a huge, general purpose knowledge graph and it gets away with ~1B triples, give or take.
Almost all analytic graphs of general scope surpass 1T edges, see below. DARPA also has an unfilled objective for 1B edge real-time continuously updated operational graphs. These are smaller and the write throughput requirements are in line with non-graph analytical databases but graph databases struggle to meet that standard.
There are countless smaller graphs for narrow domains that may be The Semantic Web companies in the 2000s had graphs that were 100B+ edges. They wanted to go much larger but hit hard scaling walls around that point. That scaling wall killed them.
Classic mapping data models are typically 10-100B edges. These could be much, much larger if they could process all the data available to them.
Of course, intelligence agencies had all kinds of graphs far beyond trillions of edges 20 years ago. People, places, things, events.
Any type of spatiotemporal entity graphs with large geographic scope are quadrillions of edges. It isn’t just a lot of inferred relationships between entities, the relationships evolve over time which also must be captured. These are probably the most commercially valuable type of graph. You could build hundreds of different graphs of this type with 1T+ edges in most regions, never mind doing it at scale. These are so large that we usually don’t store them. Subgraphs are generated on demand, which is computationally expensive.
These spatiotemporal entity graphs also have the largest write loads. Single sources generate tens of PB/day of new edges. There is a ton of industrial data that looks like this; it isn’t just people slinging structured data.
Graphs are everywhere but we furiously avoid them because the scalability of operations over anything but severely constrained graphs is so poor. Selection bias.
NSA in particular heavily funded foundational theoretical and applied computer science research into scaling graph computing for decades. They had all kinds of boring graphs where trillions of edges was their Tuesday. The US military also uses large graph databases in fairly boring applications that probably didn’t require a graph database.
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#70Every time I look at graph databases, I just cannot figure out what problem they're solving. Particularly in an LLM based world. Don't get me wrong, graphs have interesting properties and there's something intriguing out these dynamic, open ended queries. But, what features/products/customer journeys are people building with a graph DB. Every time I explore, I end up back at "yea, but a standard DB will do 90% of thi…
A standard DB ala Postgres will be a perfectly functional graph database unless you're doing very specialized network analysis queries, which is not what most of these "knowledge graph" databases are being used for. It's only querying and data modeling that's a bit fiddly (expressing the "graph" structure using SQL) and that's being improved by the new Property Graph Query (PGQ) in the latest SQL standards.
It'd be great if PG came with a serverless/embeddable mode, that'd be the main missing thing in comparison to this tool.
I know pglite, and while it's great someone made that, it's definitely not the same