Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust’s dependencies are starting to worry me

vincents.dev

Post reply on HN
Print view View original

151–160 of 593 posts

Re: Rust’s dependencies are starting to worry me

#151

Post by dathinab »

SamuelAdams wrote: post #134
kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

This idea is already implemented in Dotnet, with Trimming and now ahead of time compilation (AOT). Maybe other languages can learn from dotnet? https://learn.microsoft.com/en-us/dotnet/core/deploying/trim... https://learn.microsoft.com/en-us/dotnet/core/deploying/nati...

dead code elimination is a very old shoe

which get reinvented all the time, like in dotnet with "trimming" or in JS with "tree-shaking".

C/C++ compiler have been doing that since before dot net was a thing, same for rust which does that since it's 1.0 release (because it's done by LLVM ;) )

The reason it gets reinvented all the time is because while it's often quite straight forward in statically compiled languages it isn't for dynamic languages as finding out what actually is unused is hard (for fine grained code elimination) or at lest unreliable (pruning submodules). Even worse for scripting languages.

Which also brings use to one area where it's not out of the box, if you build .dll/.so in one build process and then use them in another. Here additional tooling is needed to prune the dynamic linked libraries. But luckily it's not a common problem to run into in Rust.

In general most code size problems in Rust aren't caused by too huge LOC of dependencies but by an overuse of monopolization. The problem of tons of LOC in dependencies is one of supply chain trust and review ability more then anything else.

Re: Rust’s dependencies are starting to worry me

#152

Post by rs186 »

constantcrying wrote: post #149
rs186 wrote: post #71

I once wanted to contribute to the popular swc project ( https://github.com/swc-project/swc ). I cloned the repo, ran build, and a whooping 20GB was gone from my disk. The parser itself ( https://github.com/swc-project/swc/blob/main/crates/swc_ecma... ) has over a dozen dependencies, including serde. Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight. I decided that I should leave…

I am counting 13 dependencies, the rest are internal ones. Are any of these superfluous or only needed for small edge cases? Serde seems exactly a case where you absolutely should use an external dependency. Also, repository size seems an extremely irrelevant metric.

13 > 12 so over a dozen dependencies. If you look at acorn or babel/parser, they barely have any dependency.

Repository size is directly related to how long it takes to run a build, which is extremely important if I were to contribute to the project.

> Serde seems exactly a case where you absolutely should use an external dependency.

I can't see any reason a parser has a hard dependency on a serialization library.

Re: Rust’s dependencies are starting to worry me

#153

Post by aeonik »

jiggawatts wrote: post #145
kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

A consideration that is often overlooked is that the waste accumulates exponentially! If each layer of “package abstraction” is only 50% utilised, then each layer multiplies the total size by 2x over what is actually required by the end application. Three layers — packages pulling in packages that pull their own dependencies — already gets you to 88% bloat! (Or just 12% useful code) An example of this is the new Wind…

if only we had a system that we could all operate on with a standard set of tools that would take care of shared resource access like this.

Re: Rust’s dependencies are starting to worry me

#155

Post by pclmulqdq »

delusional wrote: post #55
schmichael wrote: post #41

We need a term like “Mature” or similar for dependencies that are done. Mature dependencies have two characteristics: 1. Well defined scope 2. Infrequent changes Nomad has many of these (msgpack, envparse, cli, etc). These dependencies go years without changing so the dependency management burden rapidly approaches zero. This is an especially useful property for “leaf” dependencies with no dependencies of their own.…

I have a lot of sympathy for this viewpoint, but I also ask that we try to remind ourselves. We are asking for professionalism from hobby projects. If you want a mature protobuf implementation you should probably buy one. Expecting some guy/gal on the internet to maintain one for your for free seems ill advised.

> I have a lot of sympathy for this viewpoint, but I also ask that we try to remind ourselves. We are asking for professionalism from hobby projects.

Nobody is asking for professional quality standards from hobby projects. At best, they are asking for hobby projects to advertise themselves as such, and not as "this is a library for [x] that you can use in your stuff with the expectations of [maintenance/performance/compatibility/etc.]."

Resume-driven development seems to cause people to oversell their hobby projects as software that is ready to have external users.

> If you want a mature protobuf implementation you should probably buy one

No software is ever developed this way. For some reason, libraries are always free. Approximately nobody will buy paid libraries.

Re: Rust’s dependencies are starting to worry me

#156

Post by dathinab »

nicoburns wrote: post #90
kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

As far as I'm aware, LTO completely solves this from a binary size perspective. It will optimise out anything unused. You can still get hit from a build time perspective though.

yes, it's not a issue of code size but a issue of supply chain security/reviewability

it's also not always a fair comparison, if you include tokio in LOC counting then you surely would also include V8 LOC when counting for node, or JRE for Java projects (but not JDK) etc.

Re: Rust’s dependencies are starting to worry me

#157

Post by constantcrying »

demarq wrote: post #5

> Many call for adding more to the rust standard library much like Go This is the way.

That is a serious burden on the maintainers, it creates all kinds of different problems, especially if the functionality of the libraries assumes a certain execution environment. Rust doesn't just target x86 desktops.

Re: Rust’s dependencies are starting to worry me

#158

Post by eikenberry »

pjmlp wrote: post #132
assassinator42 wrote: post #95

Earlier quoted context omitted.

Java and the .NET Framework had partial trust/capabilities mechanisms decades ago. No one really used them and they were deprecated/removed.

It was more like no one used them correctly .

Wouldn't that mean they were poorly implemented. If no one uses something correctly, seems like that isn't a problem with the people but the thing.

Re: Rust’s dependencies are starting to worry me

#159

Post by dathinab »

tialaramex wrote: post #139
xlii wrote: post #94

Earlier quoted context omitted.

> Go and Rust, for example, encourage everything for a single package/mod to go in the same file. Clarification: Go allows for a very simple multi-file. It’s one feature I really like, because it allows splitting otherwise coherent module into logical parts.

Yeah, likewise Rust is completely fine after you say `mod foo` and have a file named foo.rs, if you also make a foo/ directory and put foo/whatever.rs and foo/something_else.rs that those are all part of the foo module. Historically Rust wanted that foo.rs to be renamed foo/mod.rs but that's no longer idiomatic although of course it still works if you do that.

to extend on this:

in rust crates are semantically one compilation unit (where in C oversimplified it's a .h/.c pair, and practically rustc will try to split it in some more units to speed up build time).

the reason I'm pointing this out is because many sources of "splitting a module across files" come from situations where 1 file is one compilation unit so you needed to have a way to split it (for organization) without splitting it (for compilation) in some sitation

Re: Rust’s dependencies are starting to worry me

#160

Post by lantastic »

meling wrote: post #80
sophacles wrote: post #62

Earlier quoted context omitted.

I have yet to come across a go project that doesn't pull in tons of 3rd party code as well. It seems like maybe you're over-stating the "culture" a bit.

Yeah, while I’ve seen some great libraries that follow the practice of minimizing their dependencies, I’m a bit annoyed with the amount of dependencies that docker will bring along [1]. I’ve been on the lookout for alternatives for my docker needs, but the state of podman, buildah and some others that I checked is similar. They all bring in roughly the same number of dependencies… if anyone knows of a stripped down G…

Wow, that's massive. I guess it's inevitable that a popular piece of open-source software for end-users will be compelled to accrue dependencies due to popular demand for features that require them.

I feel Telegraf made a good compromise: out of the box, it comes with a _ton_ of stuff[1] to monitor everything, but they make it possible to build only with pieces that you need via build tags, and even provide a tool to extract said tags from your telegraf config[2]. But lots of supply-chain security stuff assume everything in go.mod is used, so that can results in a lot of noise.

[1] https://github.com/influxdata/telegraf/blob/master/go.mod [2] https://github.com/influxdata/telegraf/tree/master/tools/cus...

Post reply on HN