Live data from Hacker News

Rewriting a high performance vector database in Rust

pinecone.io

11–20 of 157 posts

Re: Rewriting a high performance vector database in Rust

#11
post #7

Earlier quoted context omitted.

Memes typically have some basis in reality. If your project reaches a point where it could benefit from fearless concurrency or better memory control, Rust is probably your best bet at the moment. I could see huge benefits from Kafka and Cassandra being re-written in Rust.

kafka clone redpanda is written in rust ?

No. C++ and the C* (seastar) framework.

Re: Rewriting a high performance vector database in Rust

#13
post #5

Glanced through the article, and I see no comparisons on how performance of the DB is in Rust versus their current C++ implementation, no mention of if maintaining the Rust code is easier than their C++ codebase, no stats on how devs are ramping up and how it's tackling their "hard to find a dev who knows both C++ and Python well" issue.

Article also states that the switch from C++ to Rust improves "low level optimized instruction sets, memory layout, and running async tasks."

The first two are also strengths of C++, and for the third the article says that "Rust is async, and Tokio is the one of the most popular async providers ... However, it’s not great for running CPU intensive workloads, like with Pinecone." Puzzling.

Re: Rewriting a high performance vector database in Rust

#15
post #7

Earlier quoted context omitted.

Memes typically have some basis in reality. If your project reaches a point where it could benefit from fearless concurrency or better memory control, Rust is probably your best bet at the moment. I could see huge benefits from Kafka and Cassandra being re-written in Rust.

kafka clone redpanda is written in rust ?

Hmm? I'm pretty sure it's written in C++.

See also, their install dependencies script.

https://github.com/redpanda-data/redpanda/blob/dev/install-d...

Re: Rewriting a high performance vector database in Rust

#17

>In addition, it’s challenging to find developers with experience in both Python and C++ So you decided on a language that makes it even harder to find experienced developers?

Anecdotally, a lot of rust-curious people seem to know python. Projects like pyo3 help a lot as they make it much easier (= safe) to build native modules compared to C, let alone C++.

Re: Rewriting a high performance vector database in Rust

#18
Rewriting in Rust is not a meme, it's a cycle.

Before Rust became viable, rewrites were done in Go.

From the archives:

- Rewriting a large production system in Go https://news.ycombinator.com/item?id=6234736 (2013)

- How We Moved Our API From Ruby to Go https://news.ycombinator.com/item?id=9693743 (2015)

- Matrix and Riot Confirmed as the Basis for France’s Secure Instant Messenger App https://news.ycombinator.com/item?id=16938545 (2018)

- Toward Vagrant 3.0 https://news.ycombinator.com/item?id=27476676 (2021)

- I’m porting the TypeScript type checker tsc to Go https://news.ycombinator.com/item?id=30074414 (2022)

Re: Rewriting a high performance vector database in Rust

#19

Earlier quoted context omitted.

Memes typically have some basis in reality. If your project reaches a point where it could benefit from fearless concurrency or better memory control, Rust is probably your best bet at the moment. I could see huge benefits from Kafka and Cassandra being re-written in Rust.

They are rewritten in C++ already; Redpanda and ScyllaDB, respectively. Why waste the effort of rewriting it once again?

I tried in 2017 writing it in rust and found some compiler bugs. I also found compiler bugs in c++ tho to be honest, but I felt more comfortable in c++ so decided to write the first version of it in c++. The huge advantage is that storage engines in particular need to be more conservative in many dimensions and having seen success with scylla, seastar was apealing to me as a 'tried and tested' for storage systems.

Prior systems I had built with facebook folly (c++ lib) and had also written my own eventing systems in the past, but the real value is having seastar being battle tested since 2016. Largely it has been the right decision for us as redpanda for it's young age has benefited from the stability of seastar.

Re: Rewriting a high performance vector database in Rust

#20
post #3

> First of all, Python is a garbage collected language, which means it can be extremely slow for writing anything high performance at scale. I don't think garbage collection is in the top 3 causes of why Python is slow.

This might be a difference of semantics- there is a difference between garbage collection as a concept being slow and python's GIL approach. My understanding is that the GIL would almost always make the top 3 reason of why python is slow in practice - it works for a very specific single threaded execution model but can't really take advantage of modern processors.
Post reply on HN