Live data from Hacker News

Rust’s dependencies are starting to worry me

vincents.dev

361–370 of 593 posts

Re: Rust’s dependencies are starting to worry me

#361
post #314

Earlier quoted context omitted.

No they don't, PTC, Aicas, GraalVM and OpenJ9 support reflection. The others no longer matter, out of business.

You can't LTO code under the presence of reflection. You can AOT but there will always be a "cold path" where you have to interpret whatever is left.

Yet it works, thanks to additional metadata, either in dynamic compiler which effectly does it in memory, throwing away execution paths with traps to redo when required, and with PGO like metadata for AOT compilation.

And since we are always wrong unless proven otherwise,

https://www.graalvm.org/jdk21/reference-manual/native-image/...

https://www.graalvm.org/latest/reference-manual/native-image...

Re: Rust’s dependencies are starting to worry me

#362
post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

>At each level a caller might need 5% of the functionality of any given dependency. The deeper the dependency tree gets the more waste piles on. Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call, but all you did was take that one dependency to format a number.

So, what's is the compiler doing that he doesnt remove unused code?

Re: Rust’s dependencies are starting to worry me

#363

Earlier quoted context omitted.

There are things added from tine to time, but yeah, some stuff in there just feels dated at this point. I’m still hoping we can get a decently typed argparse with a modern API though (so much better for tiny scripts without deps!)

I'm thankful argparse exists in pythons stdlib. But argument parsing is not that hard especially for simpler programs. programmers should be able to think for a minute and figure it out instead of always reaching for clap, thats how you get dependency hell. Argument parsing, in partucular, is a great place to start realizing that you can implement what you need without adding a dozen dependencies

Hard disagree. Standardized flag parsing is a blessing on us all, do not want to jave to figure out what flag convention the author picked to implement of the many lile one does with non getopt c programs.

Don't disagree with the principle, there are a lot of trivial pythong deps, but rolling your own argument parsing is not the way

Re: Rust’s dependencies are starting to worry me

#364
Vendoring is a step in the right direction, you’ve constrained one side of the equation.

But you’re still open to typo squatting and similar issues like crates falling unmaintained - the article mentions the now famous dotenv vs. dotenvy issue (is this solvable with a more mature governance model for the crates ecosystem? At this point dotenv should probably be reclaimed). So after vendoring a baseline set of dependencies, you need to perform comprehensive auditing.

Maybe you can leverage LLMs to make that blob of vendored deps smaller / cheaper to own. Maybe you can distill out only the functionality you need (but at what cost, now you might struggle to backport fixes published upstream). Maybe LLMs can help with the auditing process itself.

You need a stream of notifications of upstream fixes to those vendored deps. Unfortunately in the real world the decision making will be harder than “ooh, there’s a sec fix, I should apply that”.

I always wonder why someone like JFrog don’t expand their offering to provide “trusted dependencies” or something similar. I.e. you pay to outsource that dependency governance and auditing. Xray scanning in the current product is a baby step toward the comprehensiveness I’m suggesting.

Taking a step back though, I’d be really careful not to throw the baby out with the bath water here. Rust has a fairly unique capability to compose work product from across unrelated developers thanks to its type system implementation (think about what happens with a C library, who’s responsible for freeing the memory, you or me?). Composition at scale is rusts super power, at least in terms of the productivity equation for large enterprises - in this context memory safety is not the sales pitch since they already have Java or whatever.

Re: Rust’s dependencies are starting to worry me

#365
post #94
post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

> Go and Rust, for example, encourage everything for a single package/mod to go in the same file. Clarification: Go allows for a very simple multi-file. It’s one feature I really like, because it allows splitting otherwise coherent module into logical parts.

Not just multiple files, but multiple directories. One versioned dependency (module) usually consists of dozens of directories (packages) and dozens to hundreds of files. Only newcomers from other languages create too many go.mod files when they shouldn't.

Re: Rust’s dependencies are starting to worry me

#366
post #361

Earlier quoted context omitted.

You can't LTO code under the presence of reflection. You can AOT but there will always be a "cold path" where you have to interpret whatever is left.

Yet it works, thanks to additional metadata, either in dynamic compiler which effectly does it in memory, throwing away execution paths with traps to redo when required, and with PGO like metadata for AOT compilation. And since we are always wrong unless proven otherwise, https://www.graalvm.org/jdk21/reference-manual/native-image/... https://www.graalvm.org/latest/reference-manual/native-image...

You do understand that the topic at hand is not shipping around all that code needed to support a trap, right?

Re: Rust’s dependencies are starting to worry me

#367

Earlier quoted context omitted.

There are things added from tine to time, but yeah, some stuff in there just feels dated at this point. I’m still hoping we can get a decently typed argparse with a modern API though (so much better for tiny scripts without deps!)

I'm thankful argparse exists in pythons stdlib. But argument parsing is not that hard especially for simpler programs. programmers should be able to think for a minute and figure it out instead of always reaching for clap, thats how you get dependency hell. Argument parsing, in partucular, is a great place to start realizing that you can implement what you need without adding a dozen dependencies

You can, but there’s always a tradeoff, as soon as I’ve added about the 3rd argument, I always wish i had grabbed a library, because i’m not getting payed to reinvent this wheel.

Re: Rust’s dependencies are starting to worry me

#368
post #355
post #348

Earlier quoted context omitted.

And from Internals discussion ( https://internals.rust-lang.org/t/add-some-form-of-precompil... ) it seems this causes more problems than it solves. It requires huge storage, for each combination of targets, and even if it is was solved some members of Rust community would see it as a step back. Me included. They are hard to audit and are step back to OSS nature of Rust.

A systems programming language is supposed to support all deployment scenarios, not to be religious.

Wdym? The language supports it, how else could have serde done it?

The issue here is getting storage and compute for build artifacs for cargo. Cargo isn't the language though.

Re: Rust’s dependencies are starting to worry me

#369
post #26

> when checking a rust security advisory mentioning that dotenv is unmaintained This is a problem with all languages and actually an area where Rust shines (due to editions). Your pulled in packages will compile as they previously did. This is not true for garbage collected languages (pun intended). > Out of curiosity I ran toeki a tool for counting lines of code, and found a staggering 3.6 million lines of rust ....…

>Your pulled in packages will compile as they previously did. This is not true for garbage collected languages (pun intended).

What do you mean?

Re: Rust’s dependencies are starting to worry me

#370
post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

>At each level a caller might need 5% of the functionality of any given dependency. The deeper the dependency tree gets the more waste piles on. Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call, but all you did was take that one dependency to format a number. So, what's is the compiler doing that he doesnt remove unused code?

A surprising amount of code might be executed in rarely-used or undocumented code paths (for example, if the DEBUG environment variable is 1 or because a plugin is enabled even if not actually used) and thus not shaken out by the compiler.
Post reply on HN