This seems hard to evaluate without a quantitative comparison to the abundance of bugs in the package ecosystems of other languages at the same age. So, for instance, how many correctness bugs existed (or, alternatively, had been found and fixed) in the Python ecosystem when Python was ten years old? The author makes a subjective claim, but from the few other languages they mention it seems they are comparing primarily to older and more stable ecosystems.
Correctness and composability bugs in the Julia ecosystem
81–90 of 419 posts
Re: Correctness and composability bugs in the Julia ecosystem
#82Earlier quoted context omitted.
I think the author addresses this. It’s a Catch-22. If you restrict use to a small subset of types you’re undermining one of Julia’s best features. As someone who has been writing a lot of numerical analysis code recently, I would absolutely love a type system that could describe and enforce numerical stability traits.
Right. It's important to remember that tools like JAX and PyTorch have total control over the numerical libraries they are differentiating, and have freedom to impose whatever semantics, rules and restrictions are convenient (immutability and referential transparency in JAX, for example). Seemingly small decisions in an existing language and library can have a big impact on the feasibility and practicality of AD.
That's exactly where Dex might improve over Julia, with language level control over mutability and effect handlers and array access safety ... time will tell.
So packages just use those features
Maybe it will hit the right trade off, or maybe Julia will adopt similar language level tools, but adjusted for dynamic semantics. Is that even possible?
Re: Correctness and composability bugs in the Julia ecosystem
#83Earlier quoted context omitted.
> Because starting with 0 is neither math nor array indexing in general. It very, very much is. Polynomials all start at a zero "index", as does just about every expansion I can think of (Fourier, Bessel, Legendre, Chebyshev, Spherical Harmonic, etc.) Combinatorics, too, make lots of use of zero indices and zero-sized sets. As for arrays, I'll leave it to Dijkstra[1] to explain why zero indexing is most natural. Zero…
> It very, very much is. Polynomials all start at a zero "index" Notice how you had to put index in quotes. Because it's not an index, it's the degree of each polynomial term, which is a power.
Notice how you're ad homenim-ing the structure of the argument and not the argument itself? I do not at all see how putting quotes around that word invalidates the argument. I did so because mathematical literature doesn't refer to it as an index (rather as a degree as you mentioned), but it very much does index each monomial. There are an infinite number of index sets for each polynomial -- just as i can index the i'th monomial, so can (i - 7), or (i - 239842), or (i - pi) -- but one of them is obviously the most natural (pun intended).
Re: Correctness and composability bugs in the Julia ecosystem
#84Earlier quoted context omitted.
Julia has more than 18k closed issues on its github. No wonder such an active user encountered a lot of it. It's not a problem with the language, though. Yes, it allows to use offsetarrays and @inbounds together, but C can read out-of-memory locations too, so what? Edit: Julia is better than C in this regard, since the usage of @inbounds is explicit, i.e. everyone can see that the code is potentially unsafe.
I think the point he was trying to make was that the example for @inbounds from the official documentation could cause out-of-bounds accesses, while it was clearly stated that you should only use @inbounds if you are sure that no out-of-bounds accesses are possible.
The issue is that there is no way to verify if OOB access is possible given an abstract type, unless you know how that type behaves, i.e. how it's indexed.
And Julia provides no way of specifying the behaviour of abstract types.
Re: Correctness and composability bugs in the Julia ecosystem
#85I've spent a lot of time developing large computational codebases in Julia, and I think the most insidious of these issues is a product of no formal way of enforcing interfaces. Using one of the common packages to build a trait system and add some sort of guarantee that all the right methods are implemented for a given trait simplifies maintenance dramatically. This doesn't catch mathematical bugs, but those crop up…
Could some of the need for interfaces be addressed by providing an extensive test battery for types of object? It seems like if something claims to be an implementation of a floating point number it should be possible to smash that type into every error ever found to uncover implementation errors.
[deleted]
Re: Correctness and composability bugs in the Julia ecosystem
#86Wait, are those examples real? I remember complaining about 1-bsaed indexing only to be told "julia is great! we have offsetindex". If it's a source of bugs, that ... greatly reduces my future interest in adopting the language.
If packages use generic indexing functions like eachindex, there would be no correctness issue with that specific example
Re: Correctness and composability bugs in the Julia ecosystem
#87Earlier quoted context omitted.
According to the article the problem is in the ecosystem, and partly the standard lib. Basically it doesn't matter if Julia the language is fine, if all the stats packages make wrong calculations. Then what is the point of Julia, if you have to rewrite all things? might as well use another language where you trust the result of the ecosystem, since it is the ecosystem you need in order to produce results.
All bugs mentioned had been quickly fixed: https://news.ycombinator.com/item?id=31397425
That comment doesn't say all bugs have been fixed, or even quickly fixed. When I check on the posted links, many are in fact still open, e.g.
https://github.com/JuliaStats/Distributions.jl/issues/1253
https://github.com/JuliaStats/StatsBase.jl/issues/642
Re: Correctness and composability bugs in the Julia ecosystem
#88I've spent a lot of time developing large computational codebases in Julia, and I think the most insidious of these issues is a product of no formal way of enforcing interfaces. Using one of the common packages to build a trait system and add some sort of guarantee that all the right methods are implemented for a given trait simplifies maintenance dramatically. This doesn't catch mathematical bugs, but those crop up…
Could some of the need for interfaces be addressed by providing an extensive test battery for types of object? It seems like if something claims to be an implementation of a floating point number it should be possible to smash that type into every error ever found to uncover implementation errors.
It's possible to hack interface verification into place at test-time, but that has a couple of problems:
1. Running the whole testing framework to determine if you implemented an interface is a high overhead when you're developing
2. You have a lot of tests to write to really check every error. Perhaps a package which defines an interface could provide a tester for this purpose
3. Interfaces should be attached to the types, and that should be sufficient for verifying the interface
I would settle for something like checking for the implementation of methods a la BinaryTraits.jl over what we have now, which is nothing. A huge step would be documentation and automated testing that proper interface methods are implemented, not even verifying if they're "correct". This drastically reduces the surface area you need to write and check to confirm compatibility with outside code.
This simple interface specification does produce design issues of its own, but correctness is much easier to handle if you know what needs to be correct in the first place.
Re: Correctness and composability bugs in the Julia ecosystem
#89It might be useful to separate the issues that are "just" bugs from the problems that come with Julia's unusual level of composability. I have no idea if Julia has more bog-standard, local bugs – things like data structure problems or compiler faults – than other languages of comparable maturity and resources, but clearly the OP has bumped into several, which is frustrating. The composition bugs – as in offsetarrays…
> That A and B can work together by default in Julia is really cool, but it also means that as you add new types and packages, you have a quadratically growing set of untested edges.
But as the authors example showed, they clearly can't work together - they just fail at runtime instead of at compile time.
Other languages have generics and interfaces to make stuff like this dynamically exchangeable. Sure, your code needs to be designed to support this, but it also means that the author explicitly thought about what they expect from their data structures. If they don't, you might suddenly find yourself violating implicit assumptions like arrays starting at 1.
Re: Correctness and composability bugs in the Julia ecosystem
#90Everything has correctness issues somewhere. Julia ships an entire patched version of LLVM to fix correctness bugs in numerical methods. It has its own implementations of things like software-side FMA because the FMA implementation of Windows is incorrect: https://github.com/JuliaLang/julia/pull/43530 . Core Julia devs are now the maintainers of things like libuv because of how much had to be fixed there. So from tho…
Good comments, Chris. I think the author has a little bit of nuance in that Julia isn't correct in the specific use cases he needs them to be. While your point is also well taken that Julia is correct in cases where other languages aren't as well. I'm a little unfamiliar with the versioning in the package ecosystem, but would you say most packages follow or enforce SemVer? Would enforcing a stricter dependency graph…
> but would you say most packages follow or enforce SemVer?
The package ecosystem pretty much requires SemVer. If you just say `PackageX = "1"` inside of a Project.toml [compat], then it will assume SemVer, i.e. any version 1.x is non-breaking an thus allowed, but not version 2. Some (but very few) packages do `PackageX = ">=1"`, so you could say Julia doesn't force SemVar (because a package can say that it explicitly believes it's compatible with all future versions), but of course that's nonsense and there will always be some bad actors around. So then:
> Would enforcing a stricter dependency graph fix some of the foot guns of using packages or would that limit composability of packages too much?
That's not the issue. As above, the dependency graphs are very strict. The issue is always at the periphery (for any package ecosystem really). In Julia, one thing that can amplify it is the fact that Requires.jl, the hacky conditional dependency system that is very not recommended for many reasons, cannot specify version requirements on conditional dependencies. I find this to be the root cause of most issues in the "flow" of the package development ecosystem. Most packages are okay, but then oh, I don't want to depend on CUDA for this feature, so a little bit of Requires.jl here, and oh let me do a small hack for OffSetArrays. And now these little hacky features on the edge are both less tested and not well versioned.
Thankfully there's a better way to do it by using multi-package repositories with subpackages. For example, https://github.com/SciML/GalacticOptim.jl is a global interface for lots of different optimization libraries, and you can see all of the different subpackages here https://github.com/SciML/GalacticOptim.jl/tree/master/lib. This lets there be a GalacticOptim and then a GalacticBBO package, each with versioning, but with tests being different while allowing easy co-development of the parts. Very few packages in the Julia ecosystem actually use this (I only know of one other package in Julia making use of this) because the tooling only recently was able to support it, but this is how a lot of packages should be going.
The upside too is that Requires.jl optional dependency handling is by far and away the main source of loading time issues in Julia (because it blocks precompilation in many ways). So it's really killing two birds with one stone: decreasing package load times by about 99% (that's not even a joke, it's the huge majority of the time for most packages which are not StaticArrays.jl) while making version dependencies stricter. And now you know what I'm doing this week and what the next blog post will be on haha. Everyone should join in on the fun of eliminating Requires.jl.