Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can a Rust binary use incompatible versions of the same library?

github.com

Post reply on HN
Print view View original

61–70 of 89 posts

Re: Can a Rust binary use incompatible versions of the same library?

#61

Post by Thaxll »

pkolaczk wrote: post #29
oefrha wrote: post #16

Earlier quoted context omitted.

Go got this right: you want an incompatible version, you have to use a different import path. Then you can only pick one version (which is deterministically the lowest possible version) for a certain import path, not a hundred different versions. Also forces people to actually take backwards compatibility seriously.

No one asked about Go here. And no, it didn’t, it’s the same PITA as in Java or maybe even worse because there are no workarounds like classloading or shading. You have no control over the transitive dependencies. The only thing you can do if there’s a conflict is asking the author to fix one of the conflicting libraries.

You can force transitive dependency to the version you want. Just do a go get and your entire project will use version x.y.z.

And I believe Go only using one version is the best solution. It avoids many problems.

Re: Can a Rust binary use incompatible versions of the same library?

#62

Post by alkonaut »

yorwba wrote: post #50
alkonaut wrote: post #45

How does this work? Assume that the log crate in its internal state has a lock it uses for synchronizing writing to some log endpoint. If I have two versions of log in my process then they must have two copies of their internal state. So they both point to the same log endpoint, but they have one mutex each ? That means it "works" at compile time but fails at runtime? That's the worst kind of "works!" Or if I depend…

If the log endpoint is external to your process and two different copies of the logging crate in the same process writing to it cause problems, two identical copies of the logging crate in different processes will likely also cause problems. The solution here is global synchronzation, not just within one process. If the log endpoint is internal to your process, how did you end up with two independent mutexes guarding…

Perhaps a log wasn't the best example due to how the resource (a log sink) is often external. Take some simpler example: a counter (such as sequential ID generator).

It's an in memory counter doing an atomic increment that returns the next ID. Two of my projects in depend on it when they create new items. Both want to generate process wide unique IDs. But if they depend on two versions of the crate then there would be two memory locations, and thus two sequences of IDs generated, so two of the frogs in my game will risk having the same ID?

There is no sharing problem here, the problem is the opposite: that there are two memory locations instead of one?

Re: Can a Rust binary use incompatible versions of the same library?

#63

Post by tialaramex »

alkonaut wrote: post #62
yorwba wrote: post #50

Earlier quoted context omitted.

If the log endpoint is external to your process and two different copies of the logging crate in the same process writing to it cause problems, two identical copies of the logging crate in different processes will likely also cause problems. The solution here is global synchronzation, not just within one process. If the log endpoint is internal to your process, how did you end up with two independent mutexes guarding…

Perhaps a log wasn't the best example due to how the resource (a log sink) is often external. Take some simpler example: a counter (such as sequential ID generator). It's an in memory counter doing an atomic increment that returns the next ID. Two of my projects in depend on it when they create new items. Both want to generate process wide unique IDs. But if they depend on two versions of the crate then there would b…

For the counter example, with what I assume was an edit change about frogs, each of the two distinct counter versions produces, as it promised, unique IDs.

If the frog numbering code is using either counter, it gets unique IDs. The problem only arises if somehow it's using both of them. But why would we expect the "unique" IDs from two different pieces of software be guaranteed never to collide? Imagine instead of counter 1.2.3 and counter 4.5.6 we've got bob_counter and cathy_counter, are you still astonished that they might give out the same IDs? Neither mentions the other, perhaps they're the same code, copy-pasted by egomaniacs.

Re: Can a Rust binary use incompatible versions of the same library?

#64

Post by galangalalgol »

MrJohz wrote: post #22
btilly wrote: post #15

This is great for avoiding conflicts when you try to get your project running. It sucks when there is a vulnerability in a particular library, and you're trying to track all of the ways in which that vulnerable code is being pulled into your project. My preference is to force the conflict up front by saying that you can't import conflicting versions. This creates a constant stream of small problems, but avoids really…

cargo tree -i log@0.3.9 will show which dependencies require this particular version of log, and how they are transitively related to the main package. In this case, you would clearly see that the out-of-date dependency comes from package "b". There are equivalents for must other package managers that take this approach, and I've never found this a problem in practice. Of course, you still need to know that there's a…

And tools like cargo audit or cargo deny can check your build tree for CVE, and suggest what to update.

Re: Can a Rust binary use incompatible versions of the same library?

#66

Post by Filligree »

jmillikin wrote: post #9
woodruffw wrote: post #7

Earlier quoted context omitted.

I'm not saying this hasn't happened to you, but I'm curious: are you working with scientific Python codebases or similar? I've done Python development off and on for the last ~10 years, and I think I can count the number of times I've had transitive conflicts on a single hand. But I almost never touch scientific/statistical/etc. Python codebases, so I'm curious is this is a discipline/practice concern in different su…

I've encountered the same problem with Python codebases in the LLM / machine learning space. The requirements.txt files for those projects are full of unversioned dependencies, including Git repositories at some floating ref (such as master/HEAD). In the easy cases, digging through the PyPI version history to identify the latest version as of some date is enough to get a working install (as far as I can tell -- maybe…

That’s still the happy case. Once upon a time I spent four days chasing dependencies before reaching out to the original author, who admitted that it hasn’t actually worked in several months but he kept on editing code anyway.

The program depended on fundamentally incompatible sub-dependencies, on different major versions.

Re: Can a Rust binary use incompatible versions of the same library?

#67

Post by josephg »

extr wrote: post #27
josephg wrote: post #5

Earlier quoted context omitted.

> (The more I work in Python, the more I think that Python's approach is actually a good one ...) I've come to the opposite conclusion. I've "git cloned" several programs in both python and ruby (which has the same behaviour) only to discover that I can't actually install the project's dependencies. The larger your gemfile / requirements.txt is, the more likely this is to happen. All it takes is a couple packages in…

I don't know, I can see it both ways. I think it depends on programming context. On one hand you're right that it's annoying and a technically unnecessary gotcha, but for some Python use cases it simplifies the mental model to simply Know which version of a particular package is running. For example, pandas and numpy are IMO bad offenders for transitive dependency issues, but it's because they're used as building blo…

Yeah; you've gotta pick your poison. Either you sometimes end up with multiple copies of numpy installed, or sometimes pip errors out and will randomly, at the worst possible time, refuse to install the dependencies of your project. Having experienced both problems a lot of times, I'll take the former answer every time thankyou. I never want surprise, blocking errors to appear out of nowhere. Especially when its always the new team members who run into them. Thats horrible DX.

Having multiple copies of numpy installed isn't an emergency. Tidy it up at your leisure. Its pretty rare that you end up with multiple copies of the same library installed in your dependency tree anyway.

As for debugging - well, debugging should still work fine so long as your debugging tools treat foo-1.x.x as if it were a completely distinct package from foo-2.x.x. Nodejs and rust both handle this by installing both versions in separate directories. The stack trace names the full file path, and the debugger uses that to find the relevant file. Simple pimple. It works like a charm.

> For web dev and something like requests, it's just not as big of a deal to have a bunch of versions installed.

I think you've got it backwards. Its much more of a big deal on the web because bundle size matters. You don't want to bundle multiple 150kb timezone databases in your website.

I've also worked on multiple web projects which ran into problems from multiple versions of React being installed. Its a common problem to run into, because a lot of naively written web components directly depend on react. Then when the major version of react changes, you can end up with a webpage trying to render a component tree, where some components are created using foreign versions of react. That causes utter chaos. Thankfully, react detects this automatically and it yells at you in the console when it happens.

Re: Can a Rust binary use incompatible versions of the same library?

#68

Post by alkonaut »

tialaramex wrote: post #63
alkonaut wrote: post #62

Earlier quoted context omitted.

Perhaps a log wasn't the best example due to how the resource (a log sink) is often external. Take some simpler example: a counter (such as sequential ID generator). It's an in memory counter doing an atomic increment that returns the next ID. Two of my projects in depend on it when they create new items. Both want to generate process wide unique IDs. But if they depend on two versions of the crate then there would b…

For the counter example, with what I assume was an edit change about frogs, each of the two distinct counter versions produces, as it promised, unique IDs. If the frog numbering code is using either counter, it gets unique IDs. The problem only arises if somehow it's using both of them. But why would we expect the "unique" IDs from two different pieces of software be guaranteed never to collide? Imagine instead of co…

> are you still astonished that they might give out the same IDs?

No I think two sequences is exactly what's asked for. What I'm wondering is whether this is a warning when cargo fails to restore the crates and must use incompatible versions. I'd only be surprised if the behavior was a clean compile + a runtime crash. Because that's usually not the design chosen in Rust.

> If the frog numbering code is using either counter, it gets unique IDs

I guess this is the issue: what I'm imagining is that the counter crate has a static mutable state. That's not a problem with atomics, but it's an antipattern (And especially so in Rust). The solution of course is to NOT use static mutable state at all. Whoever wants to create either a Frog or a Wizard must pass in a universe, from which it can grab the next ID, so it's a counter instance rather than get_next_static_id().

My question isn't "do we really get two memory locations" (of course we do) my question is: how afraid should one be about this, i.e. what is the behavior of the compiler/cargo or other linters when it comes to warnings etc? Are there any non-contrived scenarios where bumping a version of a dependency causes a problem at runtime (only)?

Re: Can a Rust binary use incompatible versions of the same library?

#69

Post by pkolaczk »

Thaxll wrote: post #61
pkolaczk wrote: post #29

Earlier quoted context omitted.

No one asked about Go here. And no, it didn’t, it’s the same PITA as in Java or maybe even worse because there are no workarounds like classloading or shading. You have no control over the transitive dependencies. The only thing you can do if there’s a conflict is asking the author to fix one of the conflicting libraries.

You can force transitive dependency to the version you want. Just do a go get and your entire project will use version x.y.z. And I believe Go only using one version is the best solution. It avoids many problems.

The problem happens when the version that works with all the other libraries in the project does not exist.

And btw: you can force any transitive version in gradle/maven/npm/cargo as well, that’s not a feature unique to Go.

Re: Can a Rust binary use incompatible versions of the same library?

#70

Post by tialaramex »

alkonaut wrote: post #68
tialaramex wrote: post #63

Earlier quoted context omitted.

For the counter example, with what I assume was an edit change about frogs, each of the two distinct counter versions produces, as it promised, unique IDs. If the frog numbering code is using either counter, it gets unique IDs. The problem only arises if somehow it's using both of them. But why would we expect the "unique" IDs from two different pieces of software be guaranteed never to collide? Imagine instead of co…

> are you still astonished that they might give out the same IDs? No I think two sequences is exactly what's asked for. What I'm wondering is whether this is a warning when cargo fails to restore the crates and must use incompatible versions. I'd only be surprised if the behavior was a clean compile + a runtime crash. Because that's usually not the design chosen in Rust. > If the frog numbering code is using either c…

> What I'm wondering is whether this is a warning when cargo fails to restore the crates

What does "fails to restore the crates" mean here?

I've seen this verb "restore" used in some .NET software, but I can't see how it relates to Rust.

Post reply on HN