91–100 of 117 posts
Very interesting project. Would be curious of a comparison with memgraph. Will definitely give it to try for my knowledge graph use case.
Feel free to point me to docs / code if these are lazy questions :)
Looks very interesting, but I've seen these kind of multi-paradigm databases like Gel, Helix and Surreal and I'm not sure that any of them quite hit the graph spot. Does Helix support much of the graph algorithm world? For things like GrapgRAG. Either way, I'd be all over it if there was a python SDK witch worked with the generated types!
Shameless plug: If you're exploring graph+vector databases, check out https://github.com/Pometry/Raphtory/ — with a full Python SDK and built-in support for most common graph algorithms. It’s built in Rust with native vector support. The open-source version is in-memory, but the commercial version supports disk-based scaling (we tested it with a 3TB graph on an M1 MacBook + insert all 100x faster than existing GraphD…
Looks really interesting. A couple of questions: Can you explain how helix handles writes? What are you using for keys? UUIDs? I'm curious if you've done, or are thinking about, any optimizations here. Feel free to point me to docs / code if these are lazy questions :)
For keys we are using UUIDs, but using the v6 timestamped uuids so that they are easily lexicographically ordered at creation time. This means keys inserted into LMDB are inserted using the APPEND flag, meaning LMDB shortcuts to the rightmost leaf in its B-Tree (rather than starting at the root) and appends the new record. It can do this because the records are ordered by creation time meaning each new record is guaranteed to be larger (in terms of big-endian byte order) than the previous record.
We also store the UUIDs as u128 values for two reasons. The first is that a u128 takes up 16 bytes where as a string UUID takes up 36 bytes. This means we store 56% less data and LMDB has to decode 56% less bytes when doing code accesses.
For the outgoing/incoming edges for nodes, we store them as fixed sizes which means LMDB packs them in, removing the 8 byte header per Key-Value pair.
In the future, we are also going to separate the properties from the stored value as empty property objects still take up 8 bytes of space. We will also make it so nothing is inserted if the properties are empty.
You can see most of this in action in the storage core file: https://github.com/HelixDB/helix-db/blob/main/helixdb/src/he...
How does this scale horizontally across multiple regions. Is this something on your roadmap?
This is very cool, and right up my alley. Hesitant to try it out because of the bespoke query language for now. I wonder if you'd like to share your thoughts on GQL becoming an ISO standard? Also, have you looked into how Neptune Analytics handles vector embeddings?
I like what GQL has done. We've definitely taken inspiration from their methodologies in building our own language.
How does it compare to SurrealDB and ChromaDB?
Again, working on benchmarks so will put them here when we're done :)
Earlier quoted context omitted.
Neo4j supports vector indexes
Neo4j first of all is very slow for vectors, so if performance is something that matters for your user experience they definitely aren't a viable option. This is probably why Neo4j themselves have released guides on how to build that middleman software I mentioned with Qdrant for viable performance. Furthermore, the vectors is capped at 4k dimensions which although may be enough most of the time, is a problem for som…
Neo4j's vector index uses Lucene's HNSW implementation. So, the performance of vector search is the same as that of Lucene. It's worth noting that performance suffers when configured without sufficient memory, like all HNSW vector indexes.
>> This is probably why Neo4j themselves have released guides on how to build that middleman software I mentioned with Qdrant for viable performance.
No, this is about supporting our customers. Combining graphs and vectors in a single database is the best solution for many users - integration brings convenience, consistency, and performance. But we also recognise that customers might already have invested in a dedicated vector database, need additional vector search features we don't support, or benefit from separating graph and vector resources. Generally, integrating well with the broader data ecosystem helps people succeed.
>> Furthermore, the vectors is capped at 4k dimensions
We occasionally get asked about support for 8k vectors. But so far, whenever I've followed up with users, there doesn't seem to be a case for them. At ~32kb per embedding, they're often not practical in production. Happy to hear about use cases I've missed.
>> Also, they don't allow pre filtering which is a problem for a few people we've spoken to including Zep AI.
We support pre- and post-filtering. We're currently implementing metadata filtering, which may be what you're referring to.
>> AND, it is super memory intensive.
It's no more memory-intensive than other similar implementations. I get that different approaches have different hardware requirements. But in all cases, a misconfigured system will perform poorly.
I spent a bit of time reading up on the internals and had a question about a small design choice (I am new to DB internals, specifically as they relate to vector DBs). I notice that in your core vector type (`HVector`), you choose to store the vector data as a `Vec `. Given what I have seen from most embedding endpoints, they return `f32`s. Is there a particular reason for picking `f64` vs `f32` here? Is the addition…
thanks for the question! we chose f64 as a default for now as just to cover all cases and we believed that basic vector operations would not be our bottleneck initially. As we optimize our HNSW implementation, we are going to add support for f32 and binary vectors and drop using Vec and instead use [f64/f32; {num_dimensions}] to avoid unnecessary heap allocation!
I'd be curious if there's some benefit to the runtime-memory utilization to baking in the precision of the vector if it's known at comptime/runtime. In my own usage of vector DBs I've only ever used a single-precision (f32), and often have a single, known dimension. But if Helix is aiming for something more general purpose, then it makes sense to offer the mixing of precision and dimension in the internals.
Cheers