Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of Rust in ClickHouse

clickhouse.com

Post reply on HN
Print view View original

21–30 of 102 posts

Re: A year of Rust in ClickHouse

#21

Post by AlotOfReading »

uecker wrote: post #19
AlotOfReading wrote: post #11

Earlier quoted context omitted.

> The main issue with Rust is that is has been oversold as a panacea for safety using exaggerated arguments. I know you know this, but Rust does provide essentially complete memory and lifetime safety if you stay within the bounds of safe. Standard C/C++ tooling has no way to even reliably detect memory safety violations, let alone fix them. It's trivial to write buffer overflows that escape ASAN, and missing a singl…

Your comment is a perfect example for those exaggerated claims.

Which claims do you think are exaggerated?

Re: A year of Rust in ClickHouse

#22

Post by uecker »

AlotOfReading wrote: post #21
uecker wrote: post #19

Earlier quoted context omitted.

Your comment is a perfect example for those exaggerated claims.

Which claims do you think are exaggerated?

Let's start with this "... Rust does provide essentially complete memory and lifetime safety if you stay within the bounds of safe." Technically true, except for the fine print. In many scenarios you have to use "unsafe", e.g. to get equal performance to C in something as trivial as a matrix transpose or when interfacing to other code (cf. this article) Let's compare this to "It's trivial to write buffer overflows that escape ASAN,". The question is how difficult it is avoid writing such overflows. But by turning this around, you already made a biased and misleading statement. Or "and missing a single violation invalidates the semantic meaning of the entire program (particularly in C++)," again technically true, but a meaningless talking point. The only practical question is whether a violation leads to exploitable bug and and how difficult / costly it is to exploit. This "no semantic meaning" comment completely disregards the reality of mitigation which often are very effective. Continuing with "which means virtually all nontrivial programs in C/C++ have UB somewhere (a point we disagree on)." I actually agree with this, but most UB is irrelevant and easily mitigated, e.g. signed overflow is not an issue at all because you just tell your compiler to turn them into traps. And "only eliminating the biggest cause of safety issues." Memory safety is nowhere even close the biggest cause of safety issues in IT. For example, it is - for me personally - completely irrelevant. I never got hacked by a memory safety issue. There are entirely different things I worry about, which Rust makes harder to deal with, e.g. supply chain security.

Re: A year of Rust in ClickHouse

#23

Post by uecker »

uecker wrote: post #22
AlotOfReading wrote: post #21

Earlier quoted context omitted.

Which claims do you think are exaggerated?

Let's start with this "... Rust does provide essentially complete memory and lifetime safety if you stay within the bounds of safe." Technically true, except for the fine print. In many scenarios you have to use "unsafe", e.g. to get equal performance to C in something as trivial as a matrix transpose or when interfacing to other code (cf. this article) Let's compare this to "It's trivial to write buffer overflows th…

Maybe adding to the last point: I worry far more that some employee or student in my lab causes a disaster by downloading compromised python package than I get hacked by any memory safety issue. We do not use Rust, but Cargo would also be a massive concern. Rustup helps to destroy decades of user education that you do not download and run scripts from the internet.

I am fully able to appreciate that memory safety is important and Rust stepped up the game in mainstream programming. I think this is cool. But the exclusive and exaggerated focus on this does more harm than good. Memory safety is certainly much more important for advertisement companies such as Google to secure their mobile spying platforms than it is for me. The religious drive to push Rust everywhere to achieve a relatively modest[1] practical improvement in memory safety clearly shows that some part of the community rather naively adopted the priorities of certain tech companies at the cost of other - maybe more relevant - things.

1. I am fully able to understand the 100% guarantees that Rust can provide when sticking to safe Rust are conceptionally a fundamental step forward compared to what C provides out of the box. But this should not be misrepresented as a huge practical step forward to what can be achieved in memory safety already in C/C++ if one cares about it.

Re: A year of Rust in ClickHouse

#24

Post by AlotOfReading »

uecker wrote: post #22
AlotOfReading wrote: post #21

Earlier quoted context omitted.

Which claims do you think are exaggerated?

Let's start with this "... Rust does provide essentially complete memory and lifetime safety if you stay within the bounds of safe." Technically true, except for the fine print. In many scenarios you have to use "unsafe", e.g. to get equal performance to C in something as trivial as a matrix transpose or when interfacing to other code (cf. this article) Let's compare this to "It's trivial to write buffer overflows th…

We have very different priorities. Because I work on safety critical systems, language semantics are very important in order to enable things like formal executable semantics and certified compilers. When I sign my name to an inspection, the inspection is relative to the language semantics. Things like exploitability and even performance are pretty far down the list of concerns, unless they happen to impact functional safety.

   In many scenarios you have to use "unsafe"
I don't agree, because unsafe is a part of the language. It's widely accepted practice in C & C++ to only use carefully chosen subsets of each language and this is enforced with linters and coding guidelines. You can straightforwardly ban unsafe the same way, or review uses more carefully, etc.

    The question is how difficult it is avoid writing such overflows.
It's nigh-impossible as far as I can tell. I already use formal methods, sanitizers, testing, static analysis, careful design, valgrind, intensive reviews, MISRA, etc. I can still quickly find new issues by firing up the fuzzer or looking at another team's code. Other large projects like Chrome and Linux have thousands of competent eyes on them and still deal with these issues too. What is everyone missing?

    e.g. signed overflow is not an issue at all because you just tell your compiler to turn them into traps.
Leaving aside the unnecessarily hyperbolic point enabling traps for my systems might literally kill someone, traps usually aren't a well supported operational mode. GCC's ftrapv is broken for example, ubsan isn't recommended for production, and GCC doesn't implement ubsan-minimal. MSVC doesn't support overflow traps at all, nor do most certified compilers. "Just use clang" obviously isn't what you're intending here, so I'm unsure how to interpret this.

In regards to memory safety being the biggest issue, I'm referencing the "70% of high severity bugs" numbers that have been put out by Microsoft and the Chrome teams and repeated by CISA in their memory safety roadmaps.

It's great that you haven't experienced large numbers of memory safety issues, but I can only speak to the lived experience of heartbleed and others. I see memory safety issues daily. I don't see supply chain attacks frequently and given how much publicity accompanied the discovery of the XZ attack, I suspect that's true for others.

Re: A year of Rust in ClickHouse

#25

Post by simonask »

uecker wrote: post #18
IshKebab wrote: post #9

Earlier quoted context omitted.

Yes. I was thinking of this: > If you do an experiment and say "C++" anywhere on the Internet, in a minute someone will chime in and educate you about the existence of Rust. > I know examples when engineers rewrite code from Rust in Rust if they like to rewrite everything in Rust. > our engineers become too nauseous from Rust poisoning > So now they [Rust devs] can write something other than new versions of old termi…

If I try to read between the lines, I think the vibe comes from feeling / being pressured to use Rust without really seeing the point and it causing frustration in this context. I think your vibe is more weird. If people have issues with Rust, it is a "shitty attitude". While, of course, C/C++ just objectively suck, right?

I think the comment you responded to gives pretty clear reasons why the take is weird or a "shitty attitude".

Competent C++ developers are the first to admit that C++ objectively sucks. It's a bad language, but that doesn't mean there aren't good reasons to use it. Claiming that C++ is great is a weird hill to die on.

Re: A year of Rust in ClickHouse

#26

Post by dep_b »

pjmlp wrote: post #8
ChocolateGod wrote: post #5

> If you do an experiment and say "C++" anywhere on the Internet, in a minute someone will chime in and educate you about the existence of Rust. Many people see this as a problem. The response to TypeScript choosing Go over Rust was pretty gross imho, no one should be abused for choosing a language.

The actual issue wasn't Go over Rust, rather having key people responsible for C# design, on a Microsoft project, going for a Google language. While at the same time, the .NET team routinely talks about .NET image problem outside traditional Microsoft shops, which naturally decisions like this aren't helping a tiny bit.

Yeah loved the language and the IDE, but only boring Enterprise stuff built with it. Also C# got so complex, since it had to absorb every idea from F# rather than making F# a viable programming language on itself and improving interop...

At a given point after being a C# programmer for years I still encountered patterns that were completely unreadable to me.

Re: A year of Rust in ClickHouse

#27

Post by simonask »

uecker wrote: post #23
uecker wrote: post #22

Earlier quoted context omitted.

Let's start with this "... Rust does provide essentially complete memory and lifetime safety if you stay within the bounds of safe." Technically true, except for the fine print. In many scenarios you have to use "unsafe", e.g. to get equal performance to C in something as trivial as a matrix transpose or when interfacing to other code (cf. this article) Let's compare this to "It's trivial to write buffer overflows th…

Maybe adding to the last point: I worry far more that some employee or student in my lab causes a disaster by downloading compromised python package than I get hacked by any memory safety issue. We do not use Rust, but Cargo would also be a massive concern. Rustup helps to destroy decades of user education that you do not download and run scripts from the internet. I am fully able to appreciate that memory safety is…

You seem to consider convenient dependency management a security hazard, which I have always found to be a pretty weird take. It logically follows that the severe difficulty of managing dependencies in C and C++ projects is actually a security feature, or how else are we supposed to understand this opinion?

Let's not pretend that anything is better on the traditional C/C++ side, where the approach is usually one or more of:

1. Vendoring dependencies in-tree. This can result in security problems from missing out on bugfixes upstream.

2. Reinventing functionality that would otherwise be served by a dependency. This can result in security problems from much less battle-tested, buggy in-house implementation. In closed-source code, this is effectively security by obscurity.

I've seen both of these cause issues in large C++ projects.

For reference, the Rust/Cargo ecosystem contains a lot of tools and infrastructure to address supply-chain security, but it will always be a difficult problem to solve.

Re: A year of Rust in ClickHouse

#28

Post by Zefiroj »

AlotOfReading wrote: post #21
uecker wrote: post #19

Earlier quoted context omitted.

Your comment is a perfect example for those exaggerated claims.

Which claims do you think are exaggerated?

The rust language is not well-specified, and if you take rust as the language specified by the compiler, then it has many soundness bugs. So even if you stay within "safe rust", you can segfault.

The "memory safety" of rust is oversold since "safety" is not formally proven for the rust language. While anecdotally memory-related bugs seem less likely, rust without unsafe is not absolutely safe.

Re: A year of Rust in ClickHouse

#29

Post by simonask »

pjmlp wrote: post #6

I love that it goes both ways, about plus and minus of both languages, including rewriting back into C++ when it made sense, and the side joke about rewriting existing CLI tools in Rust. While C++ isn't perfect, has the warts of a 50 year's old language, and probably will never match Rust's safety, we would already be in a much better place if at least everyone used the tools at their disposal from the last 30 years.…

Coming from C++, my favorite take on Rust is that it is fundamentally about productivity.

Avoiding UB is a serious drain on productivity in C++, and every new language or library feature comes with additional pitfalls, increasing the mental load.

This is to say: The benefit of Rust is not actually about "security critical scenarios", but much more generally about delivering the same quality of code in a fraction of the time.

Re: A year of Rust in ClickHouse

#30

Post by blub »

AlotOfReading wrote: post #24
uecker wrote: post #22

Earlier quoted context omitted.

Let's start with this "... Rust does provide essentially complete memory and lifetime safety if you stay within the bounds of safe." Technically true, except for the fine print. In many scenarios you have to use "unsafe", e.g. to get equal performance to C in something as trivial as a matrix transpose or when interfacing to other code (cf. this article) Let's compare this to "It's trivial to write buffer overflows th…

We have very different priorities. Because I work on safety critical systems, language semantics are very important in order to enable things like formal executable semantics and certified compilers. When I sign my name to an inspection, the inspection is relative to the language semantics. Things like exploitability and even performance are pretty far down the list of concerns, unless they happen to impact functiona…

So you’re using formal methods, sanitizers, the whole gamut of verification and yet you can “quickly” find new issues just by fuzzing. Sounds like there’s some significant problem there that you’re not mentioning.

Microsoft and Google have a ton of legacy code, they need to have high performance because they’re pushing everything to the web in order to spy better on people, they always churn their software and they are a very juicy target. As far as I’m concerned, they should rewrite everything in Rust and stop telling other people what to do.

But of course, they also need to sell Rust to the public, otherwise they would run out of developers or would have to maintain everything themselves. Hence the cheerleading.

This blog post is much closer to the reality of using Rust in production. In fact I’d add a couple of pitfalls myself:

* original cheerleader gets bored of the Rust rewrite/moves on and the project dies.

* original cheerleader moves on and the project lives under maintenance with non-Rust programmers which do not enjoy working on it and delay and reject changes and/or feature requests.

Post reply on HN