Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust’s dependencies are starting to worry me

vincents.dev

Post reply on HN
Print view View original

141–150 of 593 posts

Re: Rust’s dependencies are starting to worry me

#141

Post by zozbot234 »

kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

> In the 80s the idea of a library of functionality was something you paid for, and painstakingly included parts of into your size constrained environment (fit it on a floppy). You probably picked apart that library and pulled the bits you needed, integrating them into your builds to be as small as possible.

If anything, the 1980s is when the idea of fully reusable, separately-developed software components first became practical, with Objective-C and the like. In fact it's a significant success story of Rust that this sort of pervasive software componentry has now been widely adopted as part of a systems programming language.

Re: Rust’s dependencies are starting to worry me

#142

Post by poincaredisk »

nicoburns wrote: post #90
kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

As far as I'm aware, LTO completely solves this from a binary size perspective. It will optimise out anything unused. You can still get hit from a build time perspective though.

"completely solves" is a bit of an overstatement. Imagine a curl-like library that allows you to make requests by URL. You may only ever use HTTP urls, but code for all the other schemas (like HTTPS, FTP, Gopher) needs to be compiled in as well.

This is an extreme example, but the same thing happens very often at a smaller scale. Optional functionality can't always be removed statically.

Re: Rust’s dependencies are starting to worry me

#143

Post by zeroxfe »

zaptheimpaler wrote: post #111

This is just a modern problem in all software development, regardless of language. We are doing more complex things, we have a much bigger library of existing code to draw from and there are many reasons to use it. Ultimately a dependency is untrusted code, and there's a long road to go in hardening entire systems to make running arbitrary dependencies safe (if its even possible). In the absence of a technical soluti…

> If it was pulled into Rust stdlib, that team would be stuck handling it, and making changes to any of that code becomes more difficult.

I think Rust really needs to do more of this. I work with both Go and Rust daily at work, Go has its library game down -- the standard library is fantastic. With Rust it's really painful to find the right library and keep up for a lot of simple things (web, tls, x509, base64 encoding, heck even generating random numbers.)

Re: Rust’s dependencies are starting to worry me

#144

Post by taeric »

dietr1ch wrote: post #107
kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

I don't think libraries are the problem, but we don't have a lot of visibility after we add a new dependency. You either take the time to look into it, or just add it and then forget about the problem (which is kind of the point of having small libraries). It should be easy to build and deploy profiling-aware builds (PGO/BOLT) and to get good feedback around time/instructions spent per package, as well as a measure o…

I agree that I don't like thinking of libraries as the problem. But they do seem to be the easiest area to point at for a lot of modern development hell. Is kind of crazy.

I'll note that it isn't just PGO/BOLT style optimizations. Largely, it is not that at all, oddly.

Instead, the problem is one of stability. In a "foundation that doesn't move and cause you to fall over" sense of the word. Consider if people made a house where every room had a different substructure under it. That, largely, seems to be the general approach we use to building software. The idea being that you can namespace a room away from other rooms and not have any care on what happens there.

This gets equally frustrating when our metrics for determining the safety of something largely discourages inaction on any dependencies. They have to add to it, or people think it is abandoned and not usable.

Note that this isn't unique to software, mind. Hardware can and does go through massive changes over the years. They have obvious limitations that slow down how rapidly they can change, of course.

Re: Rust’s dependencies are starting to worry me

#145

Post by jiggawatts »

kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

A consideration that is often overlooked is that the waste accumulates exponentially!

If each layer of “package abstraction” is only 50% utilised, then each layer multiplies the total size by 2x over what is actually required by the end application.

Three layers — packages pulling in packages that pull their own dependencies — already gets you to 88% bloat! (Or just 12% useful code)

An example of this is the new Windows 11 calculator that can take several seconds to start because it loads junk like the Windows 10 Hello for Business account recovery helper library!

Why? Because it has currency conversion, which uses a HTTP library, which has corporate web proxy support, which needs authentication, which needs WH4B account support, which can get locked out, which needs a recovery helper UI…

…in a calculator. That you can’t launch unless you have already logged in successfully and is definitely not the “right place” for account recovery workflows to be kicked off.

But… you see… it’s just easier to package up these things and include them with a single line in the code somewhere.

Re: Rust’s dependencies are starting to worry me

#146

Post by CBLT »

SamuelAdams wrote: post #134
kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

This idea is already implemented in Dotnet, with Trimming and now ahead of time compilation (AOT). Maybe other languages can learn from dotnet? https://learn.microsoft.com/en-us/dotnet/core/deploying/trim... https://learn.microsoft.com/en-us/dotnet/core/deploying/nati...

Those are done at compile time. Many languages (including Rust, which this story is about) also remove unused symbols at compile time.

The comment you're replying to is talking about not pulling in dependencies at all, before compiling, if they would not be needed.

Re: Rust’s dependencies are starting to worry me

#147

Post by kdps »

zaptheimpaler wrote: post #111

This is just a modern problem in all software development, regardless of language. We are doing more complex things, we have a much bigger library of existing code to draw from and there are many reasons to use it. Ultimately a dependency is untrusted code, and there's a long road to go in hardening entire systems to make running arbitrary dependencies safe (if its even possible). In the absence of a technical soluti…

I'd argue that the severity varies between languages, despite the core problem being universal. Languages with comprehensive standard libraries have an advantage over those with minimal built-in functionality, where people rely on external dependencies even for the most basic things (e.g. see Java/.NET vs JS/Node). Lightweight is not always better.

Re: Rust’s dependencies are starting to worry me

#148

Post by metaltyphoon »

timewizard wrote: post #73
nemothekid wrote: post #43

I feel like leftpad has given package managers a very bad name. I understand the OP's hesitation, but it feels a little ridiculous to me. tokio is a work-stealing, asynchronous runtime. This is a feature that would be an entire language . Does OP consider it reasonable to audit the entire Go language? or the V8 engine for Node? v8 is ~10x more lines than tokio. If Cloudflare uses Node, would you expect Cloudflare to…

If two different dependencies use a different version of some other dependency between them does cargo still include both versions by default? This is something I've only ever seen cargo do.

> If two different dependencies use a different version of some other dependency between them does cargo still include both versions by default?

No, cargo will resolve using sem ver compatibility and pick the best version. Nuget, for C# does something very similar.

Re: Rust’s dependencies are starting to worry me

#149

Post by constantcrying »

rs186 wrote: post #71

I once wanted to contribute to the popular swc project ( https://github.com/swc-project/swc ). I cloned the repo, ran build, and a whooping 20GB was gone from my disk. The parser itself ( https://github.com/swc-project/swc/blob/main/crates/swc_ecma... ) has over a dozen dependencies, including serde. Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight. I decided that I should leave…

I am counting 13 dependencies, the rest are internal ones. Are any of these superfluous or only needed for small edge cases? Serde seems exactly a case where you absolutely should use an external dependency.

Also, repository size seems an extremely irrelevant metric.

Re: Rust’s dependencies are starting to worry me

#150

Post by rs186 »

MeetingsBrowser wrote: post #77
rs186 wrote: post #71

I once wanted to contribute to the popular swc project ( https://github.com/swc-project/swc ). I cloned the repo, ran build, and a whooping 20GB was gone from my disk. The parser itself ( https://github.com/swc-project/swc/blob/main/crates/swc_ecma... ) has over a dozen dependencies, including serde. Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight. I decided that I should leave…

I agree that relying on unknown dependencies is a risk, but this misses the point IMO. Number of dependencies and disk space are kind of arbitrary. > Meanwhile, the heaviest JavaScript parser implemented in JavaScript is more lightweight. The lightest weight javascript program relies on V8 to run, which has multiple orders of magnitude more dependencies. Most of which you have never heard of. At least cargo makes it…

No, it has very little to do with v8 or any runtime. Those parsers run on any decent and recent enough runtime, including browsers and Node.js. If you look at the actual code, they use basic APIs in the JavaScript language that you can find in almost any other language.
Post reply on HN