Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust’s dependencies are starting to worry me

vincents.dev

Post reply on HN
Print view View original

381–390 of 593 posts

Re: Rust’s dependencies are starting to worry me

#381

Post by ak_111 »

Dependency and build management is a fascinating and still unsolved problem in software engineering (in some sense it is the central problem).

I am wondering if there is a good modern reference that provides a conceptual overview or comparative study of the various techniques that have been attempted.

It is a hard subject to define as it cuts through several layers of the stack (all the way down to the compiler system interface layer), and most book focus on one language or build technology rather than providing a more conceptual treatment of the techniques used.

Re: Rust’s dependencies are starting to worry me

#382

Post by nosianu »

bilbo-b-baggins wrote: post #266
kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

Symbol culling and dead code removal is already a thing in modern compilers and linkers, and rust can do it too: https://github.com/johnthagen/min-sized-rust

As others have pointed out elsewhere, that only removes static dependencies. If you have code paths that are used depending on dynamic function arguments static analysis is unable to catch those.

For example, you have a function calling XML or PDF or JSON output functions depending on some output format parameter. That's three very different paths and includes, but if you don't know which values that parameter can take during runtime you will have to include all three paths, even if in reality only XML (for example) is ever used.

Or there may be higher level causes outside of any analysis, even if you managed a dynamic one. In a GUI, for example, it could be functionality only ever seen by a few with certain roles, but if there is only one app everything will have to be bundled. Similar scenarios are possible with all kinds of software, for example an analysis application that supports various input and output scenarios. It's a variation of the first example where the parameter is internal, but now it is external data not available for an analysis because it will be known only when the software is actually used.

Re: Rust’s dependencies are starting to worry me

#383

Post by ramon156 »

Charon77 wrote: post #248

One of the good thing in cargo packages are the feature flags. If a repo uses too much dependencies then it's time to open an issue or PR to hide them behind feature flags. I do that a lot with packages that requires std even though it could do with core and alloc. cargo tree helps a lot on viewing dependency tree. I forgot if it does LoC count or not.. > to see what lines ACTUALLY get compiled into the final binary,…

100%, I still miss feature flags in npm. Is there a package manager that can do this already? I'd love to expand our internal libs with framework-specific code

Re: Rust’s dependencies are starting to worry me

#384

Post by ak_111 »

robertlagrant wrote: post #84
Orangeair wrote: post #76

Earlier quoted context omitted.

It's a long standing bug, setenv and unsetenv are not thread-safe https://www.evanjones.ca/setenv-is-not-thread-safe.html

I honestly think using setenv is just a terrible idea.

can you elaborate what is the simpelist alternative?

Re: Rust’s dependencies are starting to worry me

#385

Post by amiga386 »

kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

> Its super easy to say `import foolib`, then call `foolib.do_thing()` and just start running.

It's effectively an end-run around the linker.

It used to be that you'd create a library by having each function in its own compilation unit, you'd create a ".o" file, then you'd bunch them together in a ".a" archive. When someone else is compiling their code, and they need the do_thing() function, the linker sees it's unfulfiled, and plucks it out of the foolib.a archive. For namespacing you'd probably call the functions foolib_do_thing(), etc.

However, object-orientism with a "god object" is a disease. We go in through a top-level object like "foolib" that holds pointers to all its member functions like do_thing(), do_this(), do_that(), then the only reference the other person's code has is to "foolib"... and then "foolib" brings in everything else in the library.

It's not possible for the linker to know if, for example, foolib needed the reference to do_that() just to initialise its members, and then nobody else ever needed it, so it could be eliminated, or if either foolib or the user's code will somehow need it.

> Go and Rust, for example, encourage everything for a single package/mod to go in the same file.

I can say that, at least for Go, it has excellent dead code elimination. If you don't call it, it's removed. If you even have a const feature_flag = false and have an if feature_flag { foobar() } in the code, it will eliminate foobar().

Re: Rust’s dependencies are starting to worry me

#386

Post by QuartzLibrary »

simonask wrote: post #322
jerf wrote: post #29

A true enough statement, but "Rust" is unnecessarily specific. Dependencies are getting scary in general. Supply chain attacks are no longer hypothetical, they're here and have been for a while. If I were designing a new language I think I'd be very interested in putting some sort of capability system in so I can confine entire library trees safely, and libraries can volunteer somehow what capabilities they need/offe…

I love this idea. There is some reminiscence of this in Rust, but it's opt in and based on convention, and only for `unsafe` code. Specifically, there's a trend of libraries using `#![deny(unsafe_code)]` (which will cause a compilation error if there is any `unsafe` code in the current crate), and then advertising this to their users. But there's no enforcement, and the library can still add `#[allow(unsafe_Code)]` t…

FYI: `#[forbid(_)]` cannot be bypassed by the affected code (without a never-to-be-stabilised nightly feature meant to be used only in `std` macros).

https://doc.rust-lang.org/rustc/lints/levels.html

Re: Rust’s dependencies are starting to worry me

#387

Post by j1elo »

jerf wrote: post #29

A true enough statement, but "Rust" is unnecessarily specific. Dependencies are getting scary in general. Supply chain attacks are no longer hypothetical, they're here and have been for a while. If I were designing a new language I think I'd be very interested in putting some sort of capability system in so I can confine entire library trees safely, and libraries can volunteer somehow what capabilities they need/offe…

Maybe we need a stronger culture of Sans-IO dependencies in general. To the point of pointing out and criticising like it happens with bad practices and dark patterns. A new lib (which shouldn't be used its own file access code) is announced in HN, and the first comment: "why do you do your own IO?"

Edit - note it's just tongue in cheek. Obviously libraries being developed against the public approval wouldn't be much of a good metric. Although I do agree that a bit more common culture of the Sans-IO principles would be a good thing.

Re: Rust’s dependencies are starting to worry me

#388

Post by PoignardAzur »

kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

> At each level a caller might need 5% of the functionality of any given dependency. The deeper the dependency tree gets the more waste piles on. Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call, but all you did was take that one dependency to format a number.

I'm not convinced that happens that often.

As someone working on a Rust library with a fairly heavy dependency tree (Xilem), I've tried a few times to see if we could trim it by tweaking feature flags, and most of the times it turned out that they were downstream of things we needed: Vulkan support, PNG decoding, unicode shaping, etc.

When I did manage to find a superfluous dependency, it was often something small and inconsequential like once_cell. The one exception was serde_json, which we could remove after a small refactor (though we expect most of our users to depend on serde anyway).

We're looking to remove or at least decouple larger dependencies like winit and wgpu, but that requires some major architectural changes, it's not just "remove this runtime option and win 500MB".

Re: Rust’s dependencies are starting to worry me

#389

Post by harha_ »

hedora wrote: post #244
pjmlp wrote: post #130

Earlier quoted context omitted.

Kind of true, when not using vcpkg/conan.

Don’t forget cmake. (It makes adding dependencies easy, and everything else basically impossible)

I think CMake is the perfect balance. You need to write few lines and think about few things before adding a dependency, but usually nothing too crazy. It might not work the first try but that's okay.

Re: Rust’s dependencies are starting to worry me

#390

Post by akshitgaur2005 »

kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

I am just a college student, so sorry if this is stupid, but we know that Rust compiler can detect unused code, variables, functions and all, as can IDE's for all languages, then why don't we just remove those parts? The unused code is just not compiled.
Post reply on HN