Live data from Hacker News

Rewriting Rust

josephg.com

71–80 of 410 posts

Re: Rewriting Rust

#71
post #50

One of the things that hit me when I was picking up Rust was that I felt like it had every imaginable feature one could think of - I dont know if Rust team said no to anything (yes I know they obviously must’ve done) - and yet people wanted more and more (some justifiably, others less so) as the language “felt” incomplete or that the features thatd be used by 2% of devs are totally necessary in the language that is “…

I wish Zig had a borrow checker... then we could see how much better it would fare.

(This is not a diss on Zig at all, I love its approach!)

Re: Rewriting Rust

#72
post #42

Earlier quoted context omitted.

That's the usual response I get when I bring this issue up. "file watching is actually very complicated" or "if you avoided deps, you'd just reimplement millions of loc yourself. Forgive me if I'm making a very bold claim, but I think cross-platform file watching should not require this much code. It's 32x larger than the Linux memory management subsystem.

Eh. The standard library is also a gigantic dependency written entirely by volunteers.

I don't have a problem with dependencies in principle. There's a good reason for standard libraries to contain a decent amount of code. It is a vector for supply chain attacks, but I also have a lot of trust in the Rust maintainers. The Rust standard library is exceptionally well-written from what I've seen, so I'm not too worried about it.

FWIW I checked out the nightly toolchain, and it looks like the stdlib is less than 400k SLoC. So literally 10x smaller.

Re: Rewriting Rust

#73

Earlier quoted context omitted.

This is the main reason we have banned Rust across my Org. Every third party library needs to be audited before being introduced as a vendored dependency which is not easy to do with the bloated dependency chains that Cargo promotes.

How do you solve this for other languages you use?

Our main languages are Go and OCaml. We can leverage third party libraries without easily running into transitive dependency hell as there’s an implicit understanding in these communities that large number of dependencies is not a good thing. Or, expressed differently, there is coarser granularity in what ends up being a library. This is not the case with Cargo which has decided to follow the NPM approach.

Re: Rewriting Rust

#74
post #42

Earlier quoted context omitted.

The fact is that dependency jungle is the prevalent way to get shit done these days. The best the runtime can do is embrace it, make it as performant and safe as possible and try to support minimum-dependency projects by having a broad std library. Also I am no expert, but I think file-watchers are definitely not simple at all, especially if they are multi-platform.

That's the usual response I get when I bring this issue up. "file watching is actually very complicated" or "if you avoided deps, you'd just reimplement millions of loc yourself. Forgive me if I'm making a very bold claim, but I think cross-platform file watching should not require this much code. It's 32x larger than the Linux memory management subsystem.

I think the issue of file-watching is that the libs usually support multiple implementations (with different tradeoffs and with multiple fallbacks) for file-watching with a lot of them being platform specific.

Re: Rewriting Rust

#75
> Rust doesn't have syntax to mark a struct field as being in a borrowed state.

> ast_nodes: Vec,

Oh, that would be neat to replace the https://github.com/tommie/incrstruct I wrote for two-phase initialization. Unlike Ouroboros and self_cell, it uses traits so the self-references can be recreated after a move. Whether it's a good idea, I don't know, but the magic Ouroboros applies to my struct feels wrong. But I say that as someone coming from C++.

> if let Some(x) = some_var && some_expr { }

Coming from Go, I was surprised that something like

    if let Some(x) = some_var; expr(x) { }
isn't a thing.

Re: Rewriting Rust

#76
post #50

One of the things that hit me when I was picking up Rust was that I felt like it had every imaginable feature one could think of - I dont know if Rust team said no to anything (yes I know they obviously must’ve done) - and yet people wanted more and more (some justifiably, others less so) as the language “felt” incomplete or that the features thatd be used by 2% of devs are totally necessary in the language that is “…

I'm curious, what drama in the Rust community are you referring to? I see some drama associated with Rust, but it's usually around people resisting its usage or adoption (the recent kerfuffle about Rust for Linux, for example), and not really that common within the community. But I could be missing something? Zig is great, but it just isn't production ready.

On drama: https://users.rust-lang.org/t/why-is-there-so-much-mismanage...

Also, Zig is set to release 1.0 beta in November.

Re: Rewriting Rust

#77
post #15

I think the dependency situation is pretty rough, and very few folks want to admit it. An example I recently stumbled upon: the cargo-watch[0] crate. At its core its a pretty simple app. I watches for file changes, and re-runs the compiler. The implementation is less than 1000 lines of code. But what happens if I vendor the dependencies? It turns out, the deps add up to almost 4 million lines of Rust code, spread acr…

The fact is that dependency jungle is the prevalent way to get shit done these days. The best the runtime can do is embrace it, make it as performant and safe as possible and try to support minimum-dependency projects by having a broad std library. Also I am no expert, but I think file-watchers are definitely not simple at all, especially if they are multi-platform.

> try to support minimum-dependency projects by having a broad std library.

Since everyone depends on the standard library this will just mean everyone will depend on even more lines of code. You are decreasing the number of nominal dependencies but increasing of much code those amount to.

Moreover the moment the stdlib's bundled dependency is not enough there are two problems:

- it can't be changed because that would be a breaking change, so you're stuck with the old bad implementation;

- you will have to use an alternative implementation in another crate, so now you're back at the starting situation except with another dependency bundled in the stdlib.

Just look at the dependency situation with the python stdlib, e.g. how many versions of urllib there are.

Re: Rewriting Rust

#78
post #42

Earlier quoted context omitted.

The fact is that dependency jungle is the prevalent way to get shit done these days. The best the runtime can do is embrace it, make it as performant and safe as possible and try to support minimum-dependency projects by having a broad std library. Also I am no expert, but I think file-watchers are definitely not simple at all, especially if they are multi-platform.

That's the usual response I get when I bring this issue up. "file watching is actually very complicated" or "if you avoided deps, you'd just reimplement millions of loc yourself. Forgive me if I'm making a very bold claim, but I think cross-platform file watching should not require this much code. It's 32x larger than the Linux memory management subsystem.

Good file watching that provides flexible primitives absolutely requires:

- ok, a single ext4 file inode changes, and its filename matches my hardcoded string

- oh, you don’t want to match against just changes to “package.json” but you want to match against a regex? voila, now you need a regex engine

- what about handling a directory rename? should that trigger matches on all files in the renamed directory?

- should the file watcher be triggered once per file, or just every 5ms? turns out this depends on your use case

- how do symlinks fit into this story?

- let’s say i want to handle once every 5ms- how do i actually wait for 5ms? do i yield the thread? do i allow other async contexts to execute while i’m waiting? how do those contexts know when to execute and when to yield back to me? now you have an async runtime with timers

- how does buffering work? are there limits on how many file change events can be buffered? do i dynamically allocate more memory as more file changes get buffered? now you need a vector/arraylist implementation

And this is before you look at what this looks like on different platforms, or if you want polling fallbacks.

Can you do it with less dependencies? Probably, if you start making hard tradeoffs and adding even more complexity about what features you activate - but that only adds lines of code, it doesn’t remove them.

What you describe is ideologically nice, but in practice it’s over-optimizing for a goal that most people don’t really care about.

Re: Rewriting Rust

#79
post #15

I think the dependency situation is pretty rough, and very few folks want to admit it. An example I recently stumbled upon: the cargo-watch[0] crate. At its core its a pretty simple app. I watches for file changes, and re-runs the compiler. The implementation is less than 1000 lines of code. But what happens if I vendor the dependencies? It turns out, the deps add up to almost 4 million lines of Rust code, spread acr…

I consciously remove and rewrite various dependencies at work, but I feel it's only a half of the whole story because either 1K or 4M lines of code seem to be equally inaccurate estimates for the appropriate number of LoC for this project.

It seems that most dependencies of cargo-watch are pulled from three direct requirements: clap, cargo_metadata and watchexec. Clap would pull lots of CLI things that would be naturally platform-dependent, while cargo_metadata will surely pull most serde stuffs. Watchexec does have a room for improvement though, because it depends on command-group (maintained in the same org) which unconditionally requires Tokio! Who would have expected that? Once watchexec got improved on that aspect however, I think these requirements are indeed necessary for the project's goal and any further dependency removal will probably come with some downsides.

A bigger problem here is that you can't easily fix other crates' excessive dependencies. Watchexec can be surely improved, but what if other crates are stuck at the older version of watchexec? There are some cases where you can just tweak Cargo.lock to get things aligned, but generally you can't do that. You have to live with excessive and/or duplicate dependencies (not a huge problem by itself, so it's default for most people) or work around with `[patch]` sections. (Cargo is actually in a better shape given that the second option is even possible at all!) In my opinion there should be some easy way to define a "stand-in" for given version of crate, so that such dependency issues can be more systematically worked around. But any such solution would be a huge research problem for any existing package manager.

Re: Rewriting Rust

#80
post #66

Earlier quoted context omitted.

Is there a known ratio of crates that use unsafe to ones that don't? It feels like most nontrivial crates would often need some unsafe. But a system like this might create a scenario where crates offload some of their unsafe code into separate crates so they need updating less frequently (Much like the blah-sys versus blah crates).

I suppose this depends on your definition of "nontrivial", but I don't think most would, unless you count the fact that some stuff in std is implemented with unsafe under the hood. The only times I've ever needed to use unsafe Rust code in 5~ years of writing it professionally was for interfacing with a vendor-specific C library, and that was only for the wrapper around it; the rest of the code didn't need to use uns…

Yes I'm probably biased towards seeing more unsafe as a deal with a lot of wrapper libs (crates which in turn have -sys crates and so on). Looking at the dependency graph, if I use 10 deps directly and 5 have unsafe then that might be 50% of the direct dependencies, but probably just a small fraction of the total including transitive.
Post reply on HN