Cross-platform Rust rewrite of the GNU coreutils • Hacker News Board

Re: Cross-platform Rust rewrite of the GNU coreutils

#431

Post by dbaupp » Wed, Mar 23, 2016, 2:59 AM UTC

collinmanderson wrote: post #425

pcwalton wrote: post #95

Earlier quoted context omitted.

Disappointing to see Hipp make that argument. It's trivially refuted. Yes, all programming languages allow the programmer to write bugs. But languages very much vary in how many , and what kinds of bugs programmers write in practice . Saying "well, Rust doesn't eliminate all bugs" is attacking a straw man. If you want to argue that Rust isn't worth it, you need to convince me that C plus gcov results in fewer bugs in…

I'm new to Rust. It is possible for a program written in C to link to a library written in Rust? Is it theoretically possible to rewrite Sqlite in Rust and still maintain compatibility with programs written in C?

Yeah, it's one of Rust's strongest features, as it allows progressive migration of native code, and also allows Rust to serve as a fast/low-overhead extension language for Ruby and Python (etc.).

More details:

- http://doc.rust-lang.org/book/ffi.html#calling-rust-code-fro... - http://doc.rust-lang.org/1.6.0/book/rust-inside-other-langua...

Re: Cross-platform Rust rewrite of the GNU coreutils

#432

Post by Dylan16807 » Wed, Mar 23, 2016, 3:08 AM UTC

jerf wrote: post #423

Dylan16807 wrote: post #407

Earlier quoted context omitted.

I still don't understand. Why would length be undefined if that's how you tell whether a string is small or not? Even if you can remove one of the branches because you know if the string is small, the logic of "this branch can't happen" -> undefined -> delete sounds more complex than "this branch can't happen" -> delete.

The order is undefined -> "I, the compiler, declare this branch can't happen" -> delete. The middle step is valid because "undefined behavior" permits the compiler to make that declaration, then act on it. If you don't want it to do that, use defined behaviors only, which is a great deal easier said than done. Partially because of how hard it is to avoid it in your own code, and partially because it is shot through a…

I just don't understand how "std::string of constant size" leads to the compiler inferring that a variable is undefined along a particular code path. In particular, on the not-taken code path, what is the uninitialized variable being branched on?

Edit: Is it figuring out that the pointer in the string object is uninitialized? That doesn't seem any easier than reasoning about the length value, and I don't see how it would lead to "br undef".

Re: Cross-platform Rust rewrite of the GNU coreutils

#433

Post by pcwalton » Wed, Mar 23, 2016, 3:11 AM UTC

kbenson wrote: post #371

pcwalton wrote: post #237

Earlier quoted context omitted.

That's not the worst of it: the truly intractable part is preventing use-after-free UB. The only ways to do this are (a) remove malloc from your language; (b) add a lifetime system (incompatible with all existing C libraries); (c) add a garbage collector (which most projects written in C will not accept for performance reasons).

I think that's mostly solved by changing the whole of the idea from "remove/disallow all undefined behavior" to "remove as much of the common needed undefined behavior as possible such that most programs need not really use it". Perfect is the enemy of good.

That'd be fine if use-after-free was a corner case that doesn't show up often in practice, but every single exploit at Pwn2Own this year was UAF…

Re: Cross-platform Rust rewrite of the GNU coreutils

#434

Post by Dylan16807 » Wed, Mar 23, 2016, 3:17 AM UTC

pcwalton wrote: post #426

Dylan16807 wrote: post #407

Earlier quoted context omitted.

I still don't understand. Why would length be undefined if that's how you tell whether a string is small or not? Even if you can remove one of the branches because you know if the string is small, the logic of "this branch can't happen" -> undefined -> delete sounds more complex than "this branch can't happen" -> delete.

LLVM has been doing this optimization since 2008, and it contains justification in the commit message. LLVM commit #60470: Teach jump threading some more simple tricks: 1) have it fold "br undef", which does occur with surprising frequency as jump threading iterates. ... Chris didn't cite any specific numbers in the log, but I believe him when he says it actually happens. You should be able to run "opt -O2 -debug-onl…

For the price of removing some of the rusty nails sticking out of C, I'll happily pay a few 16KB on a binary as large as clang. Plus in most or all of those cases, the optimization would still be possible even with stable uninitialized values. It just needs to be done in a different way. You might be able to get 13 of those 16KB back with a very minor amount of work.

Re: Cross-platform Rust rewrite of the GNU coreutils

#435

Post by pcwalton » Wed, Mar 23, 2016, 3:22 AM UTC

bigdubs wrote: post #133

0xFFC wrote: post #127

Earlier quoted context omitted.

>Also Rust doesn't have exceptions so you have to wrap almost any function call with let/match/Ok/Err. Ugly. respectfully, This is pure nonsense. First of all rust does not have runtime and AFAIK for providing exception you should have runtime to manage stack. Second not every language should be like high-level languages, it is not the rule to be like C#,Java,Python,etc. I use a lot of them for my work when I need si…

Comment OP's stance is valid. "Magic" in this context are keywords or symbols that are not immediately clear to programmers that don't work in rust. One of the reasons golang is so successful is that there is very little magic in the syntax, and even when there is it's fairly easy to grok (an example would be the `go` keyword). FWIW I also share their opinion that rust is unapproachable.

> One of the reasons golang is so successful is that there is very little magic in the syntax, and even when there is it's fairly easy to grok (an example would be the `go` keyword).

Do you have a specific symbol you would like to change in Rust, and what would you like to change it to?

The only example I've seen (in a child comment to yours) is effectively a complaint that Rust has lifetimes and Go doesn't, which is effectively saying "you should have a garbage collector like Go does", which is an argument against a fundamental design decision of Rust. If you want to argue that you should always use a garbage collector, argue that directly instead of making vague negative comparisons between Rust's and Go's syntax.

Re: Cross-platform Rust rewrite of the GNU coreutils

#436

Post by pcwalton » Wed, Mar 23, 2016, 3:23 AM UTC

codedokode wrote: post #139

bigdubs wrote: post #133

Earlier quoted context omitted.

Comment OP's stance is valid. "Magic" in this context are keywords or symbols that are not immediately clear to programmers that don't work in rust. One of the reasons golang is so successful is that there is very little magic in the syntax, and even when there is it's fairly easy to grok (an example would be the `go` keyword). FWIW I also share their opinion that rust is unapproachable.

Golang has some magic too - for example magic with capital letters in exported symbols. But generally golang is easier to read than line like this: > static NAME: &'static str = "du";

Because Rust has lifetimes and Go doesn't, so Go doesn't need syntax for them. If you want to argue that Rust should use a garbage collector like Golang does (which entails arguing that everybody who is using Rust is wrong for not wanting an always-on GC) I'm happy to have that argument, but say so explicitly.

Re: Cross-platform Rust rewrite of the GNU coreutils

#437

Post by nickpsecurity » Wed, Mar 23, 2016, 3:34 AM UTC

jstimpfle wrote: post #342

nickpsecurity wrote: post #206

Earlier quoted context omitted.

Let me re-read it. He makes several points. The first is that they're intentionally relying on undefined behavior known to cause problems sometimes out of nowhere. He says it's OK they rely on it because it's not currently causing them problems. Relying on something impossible to rely on as kbenson worded it since that's working out so far. Reminds me of a George Carlin quip about people building villages on active v…

> Next, he conflates compiler bugs with undefined behavior. Maybe Rust is just not specified for every corner case? The compiler could just do anything in such cases (e.g. what a C compiler would do -- not checking for arithmetic overflow, for example). You can then go ahead and claim it wasn't UB in Rust. But effectively it is the same, and you can't expect that Rust will specify that a compiler must check for arith…

He and I both agreed UB could show up in Rust although it defaults on quite a bit of safety. Far as compiler maturity, he could use that argument but counters himself with machine code verification due to no trust in compilers. As in, what was point of being up Rust compiler quality if he doesnt trust C's either.

Only valid point he has is that a system language needs tools to produce and/or test machine code output for source equivalence. Claims Rust doesnt have that but that's outside my knowledge. I know Ada, SPARK, CompCert C, and a Java subset have methods available.

Re: Cross-platform Rust rewrite of the GNU coreutils

#438

Post by quotemstr » Wed, Mar 23, 2016, 3:35 AM UTC

tatterdemalion wrote: post #205

nickpsecurity wrote: post #175

Earlier quoted context omitted.

Holy crap, I'd never have imagined that... A design that assumes allocation always succeeds flies in the face of decades of safety/security-critical coding wisdom. I strongly recommend the team revisit and change that somehow to account for failures, NULL's, whatever. Actually, same anywhere a failure-prone, esp hardware, resource is acquired. C apps can handle this issue so Rust should as well if it's to replace the…

The standard allocator will abort on oom error. For applications which need to be tolerant to oom errors, you need to use a different allocator. Most programs can't tolerate oom errors, though, and it would be absolutely unreasonable for every function in the standard library that might perform an allocation to return an error that has to be handled by every programmer all the time. EDIT: C's solution is to make it v…

Every function in the C++ standard library reliably communicates allocation failure to the application without relying on aborting the whole program. If Rust can do it, C++ can do it too. Rust got itself into this trap by eschewing exceptions.

Re: Cross-platform Rust rewrite of the GNU coreutils

#439

Post by kazinator » Wed, Mar 23, 2016, 3:50 AM UTC

dikaiosune wrote: post #248

frutiger wrote: post #235

Earlier quoted context omitted.

I'm not going to comment about cygwin (because its build process is not as transparent) but instead about msys2, the spiritual successor to cygwin. See for yourself: https://github.com/Alexpux/MSYS2-packages/blob/master/coreut... is the `pacman` style `PKGBUILD` used to build the `coreutils` package.

That looks pretty clean, but they're still applying looks like 5 patches which would need to be maintained separately. It's certainly not as bad as a full port, but it's still not the same codebase. Rust's cross-platform efforts would (I think?) obviate that process.

But, nevertheless, it's not exactly a port of Coreutils to Windows per se.

For instance, let's take the "cp" utility.

A Windows port "cp" should be able to copy Windows-specific file attributes. Does the Cygwin "cp" do that? How about NTFS forks?

I don't believe that you can write some portable code in Rust, and have a "cp" utility which correctly handles every quirk of every OS's file structure.

Re: Cross-platform Rust rewrite of the GNU coreutils

#440

Post by pcwalton » Wed, Mar 23, 2016, 3:55 AM UTC

Dylan16807 wrote: post #434

pcwalton wrote: post #426

Earlier quoted context omitted.

LLVM has been doing this optimization since 2008, and it contains justification in the commit message. LLVM commit #60470: Teach jump threading some more simple tricks: 1) have it fold "br undef", which does occur with surprising frequency as jump threading iterates. ... Chris didn't cite any specific numbers in the log, but I believe him when he says it actually happens. You should be able to run "opt -O2 -debug-onl…

For the price of removing some of the rusty nails sticking out of C, I'll happily pay a few 16KB on a binary as large as clang. Plus in most or all of those cases, the optimization would still be possible even with stable uninitialized values. It just needs to be done in a different way. You might be able to get 13 of those 16KB back with a very minor amount of work.

> For the price of removing some of the rusty nails sticking out of C, I'll happily pay a few 16KB on a binary as large as clang.

Lots of LLVM users won't. The competition between LLVM and GCC is (or was) pretty brutal.

> Plus in most or all of those cases, the optimization would still be possible even with stable uninitialized values. It just needs to be done in a different way.

I doubt that's possible without justification. How? Are you familiar with all of the passes of LLVM, how they interact, and with the code patterns generated by the STL?

I trust Chris in that he didn't add the jump threading optimization for no reason, which is the main one that matters here. If he says that this occurs with "surprising frequency", sorry, but I'm going to trust him. Compiler developers are usually right about the impact of their optimizations. Submit a patch to LLVM if you like to remove it, but I highly doubt it'll go through. If it did go through, Rust (and perhaps Swift) would probably revert it, as we ensure that we don't emit UB in the front end, so losing the optimization hurts us for no reason.