Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rewriting a high performance vector database in Rust

pinecone.io

Post reply on HN
Print view View original

61–70 of 157 posts

Re: Rewriting a high performance vector database in Rust

#61

Post by krona »

pornel wrote: post #51
kaladin_1 wrote: post #41

I have no problem with people rewriting their projects in whatever language they see fit. What stood out for me in the article is him saying that it's difficult getting developers with experience in both Python and C++. So, I wonder, if his in-house devs could pick up Rust that they previously couldn't write, why does he think he can not hire a good programmer and charge him to learn the stack the company uses. Why m…

Despite Rust's steep learning curve, it's also paradoxically easy to add novice Rust programmers to a project. This is because inexperienced Rust programmers are relatively harmless. Noob mistakes won't compile, rather than running into dangerous gotchas. You can tell noobs not to use `unsafe` (and there are ways to enforce that), and mostly they'll just write inefficient or non-idiomatic code, but the code will be f…

Memory sanitizers, address sanitizers, leak sanitizers, threading sanitizers, undefined behaviour sanitizers. The visual studio core guidelines checker. The clang-tidy core guideline checker. I could go on but my point is, the landscape does not really look like how you've painted it.

Re: Rewriting a high performance vector database in Rust

#62

Post by rastignack »

jhgg wrote: post #44
rastignack wrote: post #34

Earlier quoted context omitted.

Where are the production grade and pure rust tls library ? Key-value store ? Ldap client ? SSH client ?

> production grade and pure rust tls library You mean rustls? https://github.com/rustls/rustls

Pure rust ? No.

Re: Rewriting a high performance vector database in Rust

#63

Post by rastignack »

howinteresting wrote: post #58
rastignack wrote: post #34

Earlier quoted context omitted.

Where are the production grade and pure rust tls library ? Key-value store ? Ldap client ? SSH client ?

I haven't used them much, but sled, ldap3 and thrussh do exist. As Rust gains further in popularity I'd expect more of these to become production ready. Meanwhile there's always C and C++ interop.

Sled and thrussh are not production grade. I don’t particularly want to delve into details as I think the effort is laudable. I can explain my position in private if need be.

Re: Rewriting a high performance vector database in Rust

#64

Post by abc_lisper »

krona wrote: post #61
pornel wrote: post #51

Earlier quoted context omitted.

Despite Rust's steep learning curve, it's also paradoxically easy to add novice Rust programmers to a project. This is because inexperienced Rust programmers are relatively harmless. Noob mistakes won't compile, rather than running into dangerous gotchas. You can tell noobs not to use `unsafe` (and there are ways to enforce that), and mostly they'll just write inefficient or non-idiomatic code, but the code will be f…

Memory sanitizers, address sanitizers, leak sanitizers, threading sanitizers, undefined behaviour sanitizers. The visual studio core guidelines checker. The clang-tidy core guideline checker. I could go on but my point is, the landscape does not really look like how you've painted it.

Yeah, but Rust guarantees sanity by design. Sanitizers are a patch, and hence not comprehensive.

Re: Rewriting a high performance vector database in Rust

#65

Post by marcosdumay »

jerf wrote: post #32
howinteresting wrote: post #30

Earlier quoted context omitted.

This is just plain false. C++ in the 1990s had nothing like serde for example.

The minimum bar for a language has moved up significantly since the 1990s. It isn't enough to just have a neat new idea, you need to ship with nearly-best-of-breed JSON serialization, a web server, a huge standard library with not just strings but things like compression and a lot of networking, and a laundry list of other things (give or take a few things) just to make it to the "barely viable alternate choice" poin…

Well, Rust checks all of those boxes.

Yes, it's a problem for that language you plan on creating (try specializing into a niche). But it's not something that should impact Rust's adoption.

Re: Rewriting a high performance vector database in Rust

#66

Post by lijogdfljk »

krona wrote: post #61
pornel wrote: post #51

Earlier quoted context omitted.

Despite Rust's steep learning curve, it's also paradoxically easy to add novice Rust programmers to a project. This is because inexperienced Rust programmers are relatively harmless. Noob mistakes won't compile, rather than running into dangerous gotchas. You can tell noobs not to use `unsafe` (and there are ways to enforce that), and mostly they'll just write inefficient or non-idiomatic code, but the code will be f…

Memory sanitizers, address sanitizers, leak sanitizers, threading sanitizers, undefined behaviour sanitizers. The visual studio core guidelines checker. The clang-tidy core guideline checker. I could go on but my point is, the landscape does not really look like how you've painted it.

They painted it like reality, though, no?

You seem to paint the landscape as full of tools and imply that they're used. Either they're insufficient or they're often under utilized, simply due to the number of bugs we see. No?

Re: Rewriting a high performance vector database in Rust

#67

Post by pjmlp »

howinteresting wrote: post #30
pjmlp wrote: post #27

Earlier quoted context omitted.

Assuming there is even a Rust library replacement to start with. People keep forgetting C++ has 30 years of being deployed in production. Rust is 2022 is like using C++ in 1990's in terms of ecosystem.

This is just plain false. C++ in the 1990s had nothing like serde for example.

And? A drop in the ocean of libraries.

Re: Rewriting a high performance vector database in Rust

#68

Post by rastignack »

pitaj wrote: post #50
rastignack wrote: post #34

Earlier quoted context omitted.

Where are the production grade and pure rust tls library ? Key-value store ? Ldap client ? SSH client ?

Aren't the most commonly used libraries for all of those written in C, not C++? Regardless, I'm surprised you haven't heard of rustls - https://github.com/rustls/rustls

You have great c++ libraries for those. Not in pure rust though.

Re: Rewriting a high performance vector database in Rust

#69

Post by smilekzs »

8jy89hui wrote: post #55
pclmulqdq wrote: post #29

Earlier quoted context omitted.

Rust is seen as more approachable by Javascript and Python devs, so they tend to learn it more often than C or C++. It is a lot more similar to JS than C++ is.

> It is a lot more similar to JS than C++ is. That is strange as I have experienced the opposite. I've written all three languages and I've noticed that JS patterns don't translate well to Rust. Many C++ patterns translate well to Rust (albeit after a bit of borrow checker fighting). Thoughts?

Rust iterators gives you JS vibes, with gotchas mostly related to lambda captures lifetimes. Once you accept that sometimes a `collect` is the easiest way out, it feels okay at the end of the day.

Re: Rewriting a high performance vector database in Rust

#70

Post by notriddle »

nicoburns wrote: post #26
pjmlp wrote: post #22

Earlier quoted context omitted.

Which is why old timers eventually learn to just deliver with boring technology.

I honestly feel like rust is boring technology in most senses of the word. It “just works” more than almost any other technology that I’ve used. The ownership system is new and different, but that’s really the only thing.

Rust is not boring technology. There's too much ecosystem churn, and new language features are deployed too often.

C++ isn't boring technology, either. If you just want to deliver value, I'd recommend Java.

Post reply on HN