Earlier quoted context omitted.
Memes typically have some basis in reality. If your project reaches a point where it could benefit from fearless concurrency or better memory control, Rust is probably your best bet at the moment. I could see huge benefits from Kafka and Cassandra being re-written in Rust.
kafka clone redpanda is written in rust ?
Rewriting a high performance vector database in Rust
11–20 of 157 posts
Re: Rewriting a high performance vector database in Rust
#12So you decided on a language that makes it even harder to find experienced developers?
Re: Rewriting a high performance vector database in Rust
#13Glanced through the article, and I see no comparisons on how performance of the DB is in Rust versus their current C++ implementation, no mention of if maintaining the Rust code is easier than their C++ codebase, no stats on how devs are ramping up and how it's tackling their "hard to find a dev who knows both C++ and Python well" issue.
The first two are also strengths of C++, and for the third the article says that "Rust is async, and Tokio is the one of the most popular async providers ... However, it’s not great for running CPU intensive workloads, like with Pinecone." Puzzling.
Re: Rewriting a high performance vector database in Rust
#14Re: Rewriting a high performance vector database in Rust
#15Earlier quoted context omitted.
Memes typically have some basis in reality. If your project reaches a point where it could benefit from fearless concurrency or better memory control, Rust is probably your best bet at the moment. I could see huge benefits from Kafka and Cassandra being re-written in Rust.
kafka clone redpanda is written in rust ?
See also, their install dependencies script.
https://github.com/redpanda-data/redpanda/blob/dev/install-d...
Re: Rewriting a high performance vector database in Rust
#16>In addition, it’s challenging to find developers with experience in both Python and C++ So you decided on a language that makes it even harder to find experienced developers?
Re: Rewriting a high performance vector database in Rust
#17>In addition, it’s challenging to find developers with experience in both Python and C++ So you decided on a language that makes it even harder to find experienced developers?
Re: Rewriting a high performance vector database in Rust
#18Before Rust became viable, rewrites were done in Go.
From the archives:
- Rewriting a large production system in Go https://news.ycombinator.com/item?id=6234736 (2013)
- How We Moved Our API From Ruby to Go https://news.ycombinator.com/item?id=9693743 (2015)
- Matrix and Riot Confirmed as the Basis for France’s Secure Instant Messenger App https://news.ycombinator.com/item?id=16938545 (2018)
- Toward Vagrant 3.0 https://news.ycombinator.com/item?id=27476676 (2021)
- I’m porting the TypeScript type checker tsc to Go https://news.ycombinator.com/item?id=30074414 (2022)
Re: Rewriting a high performance vector database in Rust
#19Earlier quoted context omitted.
Memes typically have some basis in reality. If your project reaches a point where it could benefit from fearless concurrency or better memory control, Rust is probably your best bet at the moment. I could see huge benefits from Kafka and Cassandra being re-written in Rust.
They are rewritten in C++ already; Redpanda and ScyllaDB, respectively. Why waste the effort of rewriting it once again?
Prior systems I had built with facebook folly (c++ lib) and had also written my own eventing systems in the past, but the real value is having seastar being battle tested since 2016. Largely it has been the right decision for us as redpanda for it's young age has benefited from the stability of seastar.
Re: Rewriting a high performance vector database in Rust
#20> First of all, Python is a garbage collected language, which means it can be extremely slow for writing anything high performance at scale. I don't think garbage collection is in the top 3 causes of why Python is slow.