Why is everything "... built in Rust" trending so easily on HN?
Grafeo – A fast, lean, embeddable graph database built in Rust
51–60 of 106 posts
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#52Earlier quoted context omitted.
alternative opinion: * it is possible to write high quality software using GenAI * not using GenAI could mean project won't be competitive in current landscape
> * it is possible to write high quality software using GenAI From examine this codebase it doesn’t appear to be written carefully with AI. It looks like code that was promoted into existence as fast as possible.
sure, there are bad genAI projects and there are good genAI projects. You can remove genAI term from previous sentence.
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#53Earlier quoted context omitted.
Are you talking about Andy Pavlo bet here? https://news.ycombinator.com/item?id=29737326 Kuzu folks took some of these discussions and implemented them. SIP, ASP joins, factorized joins and WCOJ. Internally it's structured very similar to DuckDB, except for the differences noted above. DuckDB 1.5 implemented sideways information passing (SIP). And LadybugDB is bringing in support for DuckDB node tables. So the idea t…
I wasn't referring to the Pavlo bet but I would make the same one! Poor algorithm and architecture scalability is a serious bottleneck. I was part of a research program working on the fundamental computer science of high-scale graph databases ~15 years ago. Even back then we could show that the architectures you mention couldn't scale even in theory. Just about everyone has been re-hashing the same basic design for d…
I guess it all depends on the meaning of the word "handle", and what the use cases are.
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#54Why is everything "... built in Rust" trending so easily on HN?
Because Rust is an excellent language that pushes you into the "pit of success", and consequently software written in Rust tends to be fast, robust and easy to deploy.
There's no big mystery. No conspiracy or organised evangelism. Rust is just really good.
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#55Serious question: are there any actually good and useful graph databases that people would trust in production at reasonable scale and are available as a vendor or as open source? eg. not Meta's TAO
plenty of those - I've had to work with dozens of different graph databases integrating them on https://gdotv.com, save for maybe 1-2 exceptions in the list of supported databases on our website, they're all production ready and either backed by a vendor or open-source (or sometimes both, e.g. Apache AGE for Azure PostgreSQL).
There are some technologies that have been around for a long time but really flying under the radar, despite being used a lot in enterprise (e.g. JanusGraph).
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#56Earlier quoted context omitted.
I wasn't referring to the Pavlo bet but I would make the same one! Poor algorithm and architecture scalability is a serious bottleneck. I was part of a research program working on the fundamental computer science of high-scale graph databases ~15 years ago. Even back then we could show that the architectures you mention couldn't scale even in theory. Just about everyone has been re-hashing the same basic design for d…
> many millions of dollars to anyone who can demonstrate a graph database that can handle a sparse trillion-edge graph. I wonder why no one has claimed it. It's possible to compress large graphs to 1 byte per edge via Graph reordering techniques. So a trillion scale graph becomes 1TB, which can fit into high end machines. Obviously it won't handle high write rates and mutations well. But with Apache Arrow based compr…
Achieving adequate performance at 1T edges in one aspect requires severe tradeoffs in other aspects, making every implementation impractical at that scale. You touched on a couple of the key issues when I was working in this domain.
There is no single machine constraint, just the observation that we routinely run non-graph databases at similar scale on single machines without issue. It doesn't scale on in-memory supercomputers either, so the hardware details are unrelated to the problem:
- Graph database with good query performance typically has terrible write performance. It doesn't matter how fast queries are if it takes too long to get data into the system. At this scale there can be no secondary indexing structures into the graph; you need a graph cutting algorithm efficient for both scalable writes and join recursion. This was solved.
- Graph workloads break cache replacement algorithms for well-understood theory reasons. Avoiding disk just removes one layer of broken caching among many but doesn't address the abstract purpose for which a cache exists. This is why in-memory systems still scale poorly. We've known how to solve this in theory since at least the 1980s. The caveat is it is surprisingly difficult to fully reduce to practice in software, especially at scale, so no one really has. This is a work in progress.
- Most implementations use global synchronization barriers when parallelizing algorithms such as BFS, which greatly increases resource consumption while throttling hardware scalability and performance. My contribution to research was actually in this area: I discovered a way to efficiently use error correction algorithms to elide the barriers. I think there is room to make this even better but I don't think anyone has worked on it since.
The pathological cache replacement behavior is the real killer here. It is what is left even if you don't care about write performance or parallelization.
I haven't worked in this area for many years but I do keep tabs on new graph databases to see if someone is exploiting that prior R&D, even if developed independently.
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#57I added a super cheap and bad embedding database in a project that allows the agent to call a tool for searching all the content it's built, it seems to work pretty well! This way the agent doesn't need to call a bunch of list tools (which I was worried would introduce lost of data to the context), and can find things based on fuzzy search.
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#58Why is everything "... built in Rust" trending so easily on HN?
It implies high performance, reliability, and a higher degree of mastery of the developer.
(Which may not all be true, but perhaps moreso than your average project)
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#59Earlier quoted context omitted.
Agreed, there's been a literal explosion in the last 3 months of new graph databases coded from scratch, clearly largely LLM assisted. I'm having to keep track of the industry quite a bit to decide what to add support for on https://gdotv.com and frankly these days it's getting tedious.
I'm turning off my brain and using neo4j
proof that Neo4j won the popularity contest!
Re: Grafeo – A fast, lean, embeddable graph database built in Rust
#60Earlier quoted context omitted.
That is a lot of code for what appears to be a vanilla graph database with a conventional architecture. The thing I would be cautious about is that graph database engines in particular are known for hiding many sharp edges without a lot of subtle and sophisticated design. It isn't obvious that the necessary level of attention to detail has been paid here.
Are you talking about Andy Pavlo bet here? https://news.ycombinator.com/item?id=29737326 Kuzu folks took some of these discussions and implemented them. SIP, ASP joins, factorized joins and WCOJ. Internally it's structured very similar to DuckDB, except for the differences noted above. DuckDB 1.5 implemented sideways information passing (SIP). And LadybugDB is bringing in support for DuckDB node tables. So the idea t…
KuzuDB, now in [maintenance mode](https://github.com/kuzudb/kuzu). Quite annoyed about that one, was using it extensively.