Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust’s dependencies are starting to worry me

vincents.dev

Post reply on HN
Print view View original

251–260 of 593 posts

Re: Rust’s dependencies are starting to worry me

#251

Post by f_devd »

zeroxfe wrote: post #143
zaptheimpaler wrote: post #111

This is just a modern problem in all software development, regardless of language. We are doing more complex things, we have a much bigger library of existing code to draw from and there are many reasons to use it. Ultimately a dependency is untrusted code, and there's a long road to go in hardening entire systems to make running arbitrary dependencies safe (if its even possible). In the absence of a technical soluti…

> If it was pulled into Rust stdlib, that team would be stuck handling it, and making changes to any of that code becomes more difficult. I think Rust really needs to do more of this. I work with both Go and Rust daily at work, Go has its library game down -- the standard library is fantastic. With Rust it's really painful to find the right library and keep up for a lot of simple things (web, tls, x509, base64 encodi…

I disagree, as I see it Rust's core-lib should be to interact with abstract features (intrinsics, registers, memory, borrow-checker, etc), and std-lib should be to interact with OS features (net, io, threads). Anything else is what Rust excels at implementing, and putting them into stdlib would restrict the adoption of different implementations.

For example there are currently 3, QUIC (HTTP/3) implementations for rust: Quiche (Cloudflare), Quinn, S2N-QUIC (AWS). They are all spec compliant, but may use different SSL & I/O backends and support different options. 2 of them support C/C++ bindings. 2 are async, 1 is sync.

Having QUIC integrated into the stdlib wouuld means that all these choices would be made beforehand and be stuck in place permanently, and likely no bindings for other languages would be possible.

Re: Rust’s dependencies are starting to worry me

#252

Post by sethops1 »

schmichael wrote: post #41

We need a term like “Mature” or similar for dependencies that are done. Mature dependencies have two characteristics: 1. Well defined scope 2. Infrequent changes Nomad has many of these (msgpack, envparse, cli, etc). These dependencies go years without changing so the dependency management burden rapidly approaches zero. This is an especially useful property for “leaf” dependencies with no dependencies of their own.…

I feel like the Go ecosystem almost serendipitously has this built in - modules marked v0.X.Y being immature and under development, and v1 or greater being mature, keeping changes mostly down to bug fixes. I think some folks may even follow this convention!

Re: Rust’s dependencies are starting to worry me

#253

Post by uecker »

crote wrote: post #209
harha_ wrote: post #129

Earlier quoted context omitted.

Sure, there are various dependencies, but it's nothing like "cargo install crate-name". Cargo makes it so effortless to joink the dumbest dependency for the simplest thing.

On the other hand, C/C++ makes it attractive to reinvent the wheel, or vendor the dependency instead. Rather than a single well-tested implementation in the ecosystem for something like sha256, you end up with every application having its own slightly-different, mostly untested, and essentially unmaintained version. Applications still need the functionality. The need doesn't magically disappear when installing depend…

That does not help you if the bug is one of many unmaintained crates and never noticed. Linux distributions aim to make sure that C application dynamically link to the right libraries instead of vendoring the code. Then the library can be updated once. IMHO this is the only reasonable approach.

Re: Rust’s dependencies are starting to worry me

#254

Post by lmm »

A thought experiment for this writer: imagine if Tokio (and all its dependencies) were moved into the Rust standard library, so that it was more like Go. Would that make them more comfortable depending on it (not that they'd have a choice any more)? If so, why?

Re: Rust’s dependencies are starting to worry me

#256

Post by jeremiahar »

kccqzy wrote: post #202
nicoburns wrote: post #90

Earlier quoted context omitted.

As far as I'm aware, LTO completely solves this from a binary size perspective. It will optimise out anything unused. You can still get hit from a build time perspective though.

Everywhere in this thread is debating whether LTO "completely" solves this or not, but why does this even need LTO in the first place? Dead code elimination across translation units in C++ is traditionally accomplished by something like -ffunction-sections, as well as judiciously moving function implementations to the header file (inline).

Clang also supports virtual function elimination with -fvirtual-function-elimination, which AFAIK currently requires full LTO [0]. Normally, the virtual functions can't be removed because the vtable is referencing them. It's very helpful in cutting down on bloat from our own abstractions.

[0] https://clang.llvm.org/docs/ClangCommandLineReference.html#c...

Re: Rust’s dependencies are starting to worry me

#257

Post by bschwindHN »

dathinab wrote: post #151
SamuelAdams wrote: post #134

Earlier quoted context omitted.

This idea is already implemented in Dotnet, with Trimming and now ahead of time compilation (AOT). Maybe other languages can learn from dotnet? https://learn.microsoft.com/en-us/dotnet/core/deploying/trim... https://learn.microsoft.com/en-us/dotnet/core/deploying/nati...

dead code elimination is a very old shoe which get reinvented all the time, like in dotnet with "trimming" or in JS with "tree-shaking". C/C++ compiler have been doing that since before dot net was a thing, same for rust which does that since it's 1.0 release (because it's done by LLVM ;) ) The reason it gets reinvented all the time is because while it's often quite straight forward in statically compiled languages i…

> In general most code size problems in Rust aren't caused by too huge LOC of dependencies but by an overuse of monopolization

*monomorphization, in case anyone got confused

Re: Rust’s dependencies are starting to worry me

#258

Post by I_AM_A_SMURF »

cmrdporcupine wrote: post #229
sitkack wrote: post #224

Earlier quoted context omitted.

Way back when, I used to vendor all the libraries for a project (Java/Cpp/Python) into a mono repo and integrate building everything into the projects build files so anyone could rebuild the entire app stack with whatever compiler flags they wanted. It worked great, but it took diligence, it also forces you to interact with your deps in ways that adding a line to a deps file does not.

This is the default way of doing things in the monorepo(s) at Google. It feels like torture until you see the benefits, and the opposite ... the tangled mess of multiple versions and giant transitive dependency chains... agony. I would prefer to work in shops that manage their dependencies this way. It's hard to find.

I've never seen a place that does it quite like Google. Is there one? It only works if you have one product or are a giant company as it's really expensive to do.

Being able to change a dependency very deep and recompile the entire thing is just magic though. I don't know if I can ever go back from that.

Re: Rust’s dependencies are starting to worry me

#259

Post by whalesalad »

kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

Agreed it’s a problem and I can’t propose a solution other than something you’ve suggested which is referencing functions by their value (tldr hashing them) kinda like what Unison(?) proposes.

But I think the best defense against this problem at the moment is to be extremely defensive/protective of system dependencies. You need to not import that random library that has a 10 line function. You need to just copy that function into your codebase. Don’t just slap random tools together. Developing libraries in a maintainable and forward seeking manner is the exception not the rule. Some ecosystems exceed here, but most fail. Ruby and JS is probably one of the worst. Try upgrading a Rails 4 app to modern tooling.

So… be extremely protective of your dependencies. Very easy to accrue tech debt with a simple library installation. Libraries use libraries. It becomes a compounding problem fast.

Junior engineers seem to add packages to our core repo with reckless abandon and I have to immediately come in and ask why was this needed? Do you really want to break prod some day because you needed a way to print a list of objects as a table in your cli for dev?

Re: Rust’s dependencies are starting to worry me

#260

Post by Aeolun »

kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

> The only real solution I can think of to deal with this long term is ultra-fine-grained symbols and dependencies. Every function, type, and other top-level language construct needs to declare the set of things it needs to run (other functions, symbols, types, etc). When you depend on that one symbol it can construct, on demand, the exact graph of symbols it needs and dump the rest for any given library.

That’s literally the JS module system? It’s how we do tree shaking to get those bundle sizes down.

Post reply on HN