Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust’s dependencies are starting to worry me

vincents.dev

Post reply on HN
Print view View original

541–550 of 593 posts

Re: Rust’s dependencies are starting to worry me

#541

Post by dietr1ch »

taeric wrote: post #144
dietr1ch wrote: post #107

Earlier quoted context omitted.

I don't think libraries are the problem, but we don't have a lot of visibility after we add a new dependency. You either take the time to look into it, or just add it and then forget about the problem (which is kind of the point of having small libraries). It should be easy to build and deploy profiling-aware builds (PGO/BOLT) and to get good feedback around time/instructions spent per package, as well as a measure o…

I agree that I don't like thinking of libraries as the problem. But they do seem to be the easiest area to point at for a lot of modern development hell. Is kind of crazy. I'll note that it isn't just PGO/BOLT style optimizations. Largely, it is not that at all, oddly. Instead, the problem is one of stability. In a "foundation that doesn't move and cause you to fall over" sense of the word. Consider if people made a…

> Instead, the problem is one of stability. In a "foundation that doesn't move and cause you to fall over" sense of the word. Consider if people made a house where every room had a different substructure under it. That, largely, seems to be the general approach we use to building software. The idea being that you can namespace a room away from other rooms and not have any care on what happens there.

I'm not sure what the problem is here.

Are you after pinning dependencies to be sure they didn't change? Generally I want updating dependencies to fix bugs in them.

Are you after trusting them through code review or tests? I don't think there's shortcuts for this. You shouldn't trust a library, changing or not, because old bugs and new vulnerabilities make erring on both sides risky. On reviewing other's code, I think Rust helps a bit by being explicit and fencing unsafe code, but memory safety is not enough when a logic bug can ruin your business. You can't avoid testing if mistakes or crashes matter.

Re: Rust’s dependencies are starting to worry me

#542

Post by zavec »

voxgen wrote: post #97
jerf wrote: post #29

A true enough statement, but "Rust" is unnecessarily specific. Dependencies are getting scary in general. Supply chain attacks are no longer hypothetical, they're here and have been for a while. If I were designing a new language I think I'd be very interested in putting some sort of capability system in so I can confine entire library trees safely, and libraries can volunteer somehow what capabilities they need/offe…

I don't think retrofitting existing languages/ecosystems is necessarily a lost cause. Static enforcement requires rewrites, but runtime enforcement gets you most of the benefit at a much lower cost. As long as all library code is compiled/run from source, a compiler/runtime can replace system calls with wrappers that check caller-specific permissions, and it can refuse to compile or insert runtime panics if the langu…

I actually started working on a tool like that for fun, at each syscall it would walk back up the stack and check which shared object a function was from and compare that to a policy until it found something explicitly allowed or denied. I don't think it would necessarily be bulletproof enough to trust fully but it was fun to write.

Re: Rust’s dependencies are starting to worry me

#543

Post by zavec »

smj-edison wrote: post #210
zaptheimpaler wrote: post #111

This is just a modern problem in all software development, regardless of language. We are doing more complex things, we have a much bigger library of existing code to draw from and there are many reasons to use it. Ultimately a dependency is untrusted code, and there's a long road to go in hardening entire systems to make running arbitrary dependencies safe (if its even possible). In the absence of a technical soluti…

Gilad Bracha has a really interesting approach to sandboxing third party libraries: Remove imports, and do everything with dependency injection. That way if you never inject say the IO subsystem, the third party code won't be able to break out. And there's no overhead, since it's all based on capabilities. Even cooler, if you want to only expose read operations, you can wrap the IO library in another library that onl…

That sounds neat, is that newspeak?

Re: Rust’s dependencies are starting to worry me

#544

Post by zavec »

elabajaba wrote: post #268
palata wrote: post #169

Earlier quoted context omitted.

> some of the difference here is just perception due to dependencies in C/C++ being less immediately visible since they're dynamically loaded. Not in my case. I manually compile all the dependencies (either because I need to cross-compile, or because I may need to patch them, etc). So I clearly see all the transitive dependencies I need in C++. And I need a lot less than in Rust, by a long shot.

Part of the rust dependency issue is that the compiler only multithreads at the crate level currently (slowly being improved on nightly, but there's still some bugs before they can roll out the parallel compiler), so most libraries split themselves up into a ton of small crates because otherwise they just take too long to compile. edit: Also, `cargo-vet` is useful for distributed auditing of crates. There's also `car…

Can't believe I'd never heard of cargo vet before, that sounds really promising!

Re: Rust’s dependencies are starting to worry me

#545

Post by palata »

jampekka wrote: post #180
palata wrote: post #170

Earlier quoted context omitted.

> It's part of the reason why software distribution on Linux has been pushed to using containers My understanding of people distributing their software in containers is that they can't be arsed to learn how to do it properly. They would install their software and ship the entire computer if that was cost effective.

What needs to be "learned properly" is sadly a huge pile of incoherent legacy cruft that ideally wouldn't be there at all. This is not to denigrate the huge and critical effort that makes current computing possible, and that is likely unavoidable in the real world. But software distribution needs to evolve.

> What needs to be "learned properly" is sadly a huge pile of incoherent legacy cruft

I don't find it incoherent, nor huge. Unless the bar for "huge" is "anything that requires more attention than asking an LLM and copy-pasting its answer", maybe.

Re: Rust’s dependencies are starting to worry me

#546

Post by palata »

rcxdude wrote: post #419
palata wrote: post #170

Earlier quoted context omitted.

> It's part of the reason why software distribution on Linux has been pushed to using containers My understanding of people distributing their software in containers is that they can't be arsed to learn how to do it properly. They would install their software and ship the entire computer if that was cost effective.

It's not a case of 'learning to do it properly', it's a case of a huge amount of effort to deal with arbitrary differences between distros, as well as fighting with distro policies that would rather ship the software with known bugs than allow two versions of a library to exist on the system.

> it's a case of a huge amount of effort to deal with arbitrary differences between distros

That is not at all a problem for open source stuff: build your project correctly, and let distros do their job. Still, open source projects are too often doing it wrong, because nobody can be arsed to learn.

> as well as fighting with distro policies that would rather ship the software with known bugs than allow two versions of a library to exist on the system.

Sounds like if you need this, you're doing it wrong. If it's a major update (e.g. 2.3.1 to 3.0.0), it's totally possible to have a new package (say `python2` and `python3`). If your users need two versions of a library that are in the same major version (e.g. 2.3.1 and 2.5.4), then you as a developer are doing it wrong. No need to fight, just learn to do it properly.

Re: Rust’s dependencies are starting to worry me

#547

Post by smj-edison »

zavec wrote: post #543
smj-edison wrote: post #210

Earlier quoted context omitted.

Gilad Bracha has a really interesting approach to sandboxing third party libraries: Remove imports, and do everything with dependency injection. That way if you never inject say the IO subsystem, the third party code won't be able to break out. And there's no overhead, since it's all based on capabilities. Even cooler, if you want to only expose read operations, you can wrap the IO library in another library that onl…

That sounds neat, is that newspeak?

Yep! One of the many cool concepts packed in that language :)

Re: Rust’s dependencies are starting to worry me

#548

Post by yubblegum »

vacuity wrote: post #526
yubblegum wrote: post #520

So asking HN: whatever happened with OSGi? Does that architecture solve the problem, and if no, why not? https://docs.osgi.org/specification/osgi.core/7.0.0/framewor... "How OSGi Changed My Life" (2008) https://queue.acm.org/detail.cfm?id=1348594

This is my first encounter with OSGi. It seems to me that the "Lego hypothesis" reflects an increasing justified approach. The ACM Queue article mentions hot plugging and dependency injection, and a comment[0] in this thread brings up Sans IO. This also ties into capabilities, as a security measure but also an approach to modularity. The common thread is that programs should be written with a strong sense of boundari…

I used to be firmly in the component oriented camp. The reality of the matter is that the conceptual (mental) model doesn't really represent the reality of composing with reusable components.

All Lego components have the same simple standard mechanism: friction coupling using concave and convex surface elements of the component. Unix pipes are the closest thing we have to a Lego like approach and there the model of "hooking pipes of bytes from sources to sinks" actually represents what happens with the software.

With components and APIs, unless we resort to some universal baseline (such as a small finite semantic API like REST's "verbs") that basically can marshall and unmarshall any arbitrary function call ('do (func, context, in-args, out-args, out-err)' the Lego metaphor break down very quickly.

The second issue are the modalities of 'interactions' between components. So this is my first encounter with "Sans-IO" (/g) but this is just addressing the interactions issue with a fiat 'no inter-actions by components'. So Lego for software: great overall expression of desired simplicity, but not remotely effective as a generative concept and imo even possibly detrimental (as it over simplifies the problem).

Now we have 2 different pieces of software tech that somewhat have managed to arrive at component orientation: using a finite set of predefined components to build general software. One is GUI components, where a small set of visual components and operational constructs ("user-events", etc.) with structural and behavioral semantics are used to create arbitrary visual interfaces for any ~domain. The other is WWW where (REST verbs of) HTTP also provide a small finite set of 'components' (here architectural) to create arbitrary services. With both, there is the tedious and painful process of mapping domain semantics to structural components.

So we can get reusable component oriented software (ecosystems) but we need to understand (per lessons of GUIs and WebApps) that a great deal of (semantic) glue code and infrastructure is necessary, just as a lot of wiring (for GUIs) and code frameworks (for WebApps) are necessary. That is what something like OSGi brings to the table.

This then leads to the question of component boundary and granulity. With things like DCOM and JEE you have fine grained components aggregated in process boundaries. The current approach is identifying process boundary as component boundary (docker, k8, microservices) (and doing away with 'application servers' in the process).

Re: Rust’s dependencies are starting to worry me

#549

Post by [deleted] »

Aurornis wrote: post #518
kion wrote: post #85

IMO any system where taking a dependency is "easy" and there is no penalty for size or cost is going to eventually lead to a dependency problem. That's essentially where we are today both in language repositories for OSS languages and private monorepos. This is partly due to how we've distributed software over the last 40 years. In the 80s the idea of a library of functionality was something you paid for, and painsta…

> Eventually you end up in a world where your simple binary is 500 MiB of code you never actually call, It’s getting hard to take these conversations seriously with all of the hyperbole about things that don’t happen. Nobody is producing Rust binaries that hit 500MB or even 50MB from adding a couple simple dependencies. You’re also not ending up with mountains of code that never gets called in Rust. Even if my Rust b…

[deleted]

Re: Rust’s dependencies are starting to worry me

#550

Post by devnullbrain »

saagarjha wrote: post #343
f_devd wrote: post #226

Earlier quoted context omitted.

let uri: Uri = get_uri_from_stdin().parse()?; If the library is made in a modular way this is how it would typically be done. The `HTTP` may be inferred by calls further along in the function.

No, this wouldn't work. The type of the request needs to be dynamic because the user can pass in any URI.

Then they can also pass in an erroneous URI. You still need some way to deal with the ones you're not accepting.
Post reply on HN